Opening up our data using fme

Editor's Notes

• #2: Hi , I am Kamrul Kashem, or Kam for short. I work for Nottingham City Council and I’d like to thank 1Spatial for giving me an opportunity to present this burning topic of Open Data in Local Government. The title of this presentation is ‘Opening up our data using FME’ If you have any questions or queries please take a note of my email address and send them through to that or catch me during the breaks.
• #3: I’ve broken this presentation down into four sections, namely why, what, where and how? We’ll be discussing a little about the Local Government’s Transparency Code and what it means. We’ll then be looking at Open Data Nottingham which is our external-facing platform to publish the data on. We’ll look at some of the formats we are publishing in and what benefits we are seeing from the work we have done so far. As well as how Open Data Nottingham helps us meet our Freedom of Information targets by reducing the requests and officer times considerably. Finally I’ll be showing you the process we go through to publish data to Open Data Nottingham with FME. And how without FME none of this all could be possible for us with limited resources and higher pressures on Local Government.
• #4: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/408386/150227_PUBLICATION_Final_LGTC_2015.pdf So what is the local government transparency code? Since the introduction of Data.Gov.Uk in 2010 the government put in place this initiative on Local Governments. The Government’s desire was to place more power into the hands of the citizens, which meant the increase in democratic accountability by publishing datasets such as the council spend data. It would also make it easier for local people to contribute to the local decision making process help shape public services. The Local Government Transparency Code aims to make available a whole range of data from Government expenditure to spatial data. This valuable resource drives innovation by giving developers the information they need to develop applications which in turn opens new markets for local businesses.
• #5: De-loit In 2013 Deloitte published analysis for the Shakespeare Review of Public Sector Information. I’ll just go through some of the key findings by Deloitte. Firstly they concluded that the estimated economic benefit of Public Sector Information in the UK was £1.8 billion in 2012. My guess would be that this value is ever increasing with recent technological advancements and the use of social media and mobile phone application use. They estimated the social benefits as £5 billion. In the report it also stated that this figure is likely to increase as public sector information is used more widely and in more impactful ways. A transparency survey resulted in 80% saying that the main benefit in opening up local government data is external accountability. Deloitte have found that the most popular and valuable datasets are mainly geospatial data, environmental data, transport data, health data and economic data. Users of such data as it states on the slides include construction, real estate, finance and insurance, public sector, arts & entertainment and recreation sectors.
• #6: The Government endorses a five step journey to fully open format of data. We at Nottingham City Council are currently at three star, which means we make available our data as machine-readable structured data in a non-proprietary format, for example CSV or XML. But as I’ll show you next we publish our data in more formats. The Government recommends that local authorities publish data in three star format within the first six months of the Local Government Transparency Code being issued. We are currently working to get our data into four stars but as for five stars, we have not yet seen the benefits of linked data nor is there many examples of it being done to justify time and resources in an already tight Local Government budgets.
• #7: So! The part you’ve all been waiting for! Maybe… Open Data Nottingham. This is our external facing website. It was developed in-house by our web developers. I’m not a web developer myself so I can’t really go into much detail about how some of the features on the website was created, but I can tell you how the data is read by this website! As you can see there’s also a quick snapshot of the Admin panel whereby non-developers like myself can quickly go in and add new datasets, linking them back to the raw data on our servers quickly and easily. Main Page – Includes: Tags List of Formats Featured Dataset What we’ve Recently added Our recent tweets to keep users up-to-date and informed And a link to our external applications gallery. Data Catalogue So that was a quick demo of Open Data Nottingham. If you want
• #9: So what types of data are we publishing? At first we had published the mandatory datasets which were non-spatial but now we have a whole array of spatial data as you saw just now. We also publish data that will help us answer our most frequent FOI requests, such as penalty charge notices, non-domestic business rates data and council spend data. We’ll be using Open Data Nottingham to meet our INSPIRE obligations – but this a whole presentation in itself!
• #10: The benefits of Open Data Nottingham are huge for us. With resources constantly being squeezed in Local Government we need to create new ways in being efficient with our limited resources. Without FME we’d really not be able to make Open Data work but we’ll go into this a little later. Our FOI requests have reduced by quite a lot on the common requests as mentioned before. We’ve saved a lot of time in automating and improving our processes with FME. We are also beginning to see our data being used by the public to create applications and hopefully the more data we publish , this should only increase and the city as a whole will start to see the benefits in new businesses and competition.
• #11: Here are some statistics. As you can see the majority of our sessions are from Nottingham. This proves that local businesses and people are most interested in our data and hopefully it means this data is being reused to make Nottingham a better city, whether that be democratic or economic. The pie chart on the right is telling us that many of our users are new visitors. We’d like to see more of them returning to the site but that all depends on us as a council providing data that serves the interest of its local people. Hopefully with time we’ll see a change with more data being published all the time.
• #12: This was one of our biggest efficiency saving projects. Whereby we reduced the time it takes to process the redaction from Council Spend data considerably from 6 people taking 3 weeks to check 15,000 records to 1 person having to only check only 50-100 records that the FME process could not handle due to a number of issues. Another thing is, the fact we have to be transparent as a Local Government it means that we have to liaise with other teams to publish their data online but also means that we help them with their current processes in order for us to deliver timely updates on the data.
• #13: As I’ve previously stated before. Some of the benefits of handling our Open Data with FME is the reduction of the number of Freedom of Information Requests we were receiving. We’ve identified key datasets that were common requests and set up workflows whereby the data owners automatically extract data from their systems into various formats, and because FME works wonders with all types of formats it means these teams do not have to invest in external software to translate their data into a format we need. Whether that’s reading directly from Oracle tables or receiving Excel spreadsheets. FME gives everyone flexibility! In this case study we have seen FOI requests reduced on average 5-10 times per week on specific requests, such as non-domestic rates accounts, penalty charge notices and planning applications.
• #14: Here’s a list of some of our popular datasets. Recently our Polling Stations dataset has taken the lead with the most hits on the page for the last year or so. Most likely due to the European Parliament Elections in 2014 and the General Election. As you can see there is a mix of Economic data as well as transport and health.
• #15: So now, lets look at how we update our data using FME, in the practical sense. Firstly we get FME to process the data, making changes where necessary and doing what FME does best, solving problems. Secondly FME exports the data into various formats that we have decided to use on Open Data Nottingham. This data is stored on a server within an accessible folder. And finally the links that you see on the dataset page on Open Data Nottingham links directly back to the file within that folder on the server. Allowing the user to download it. That is the basic workflow in simple terms!
• #16: Here is an example of our Street Works data. Don’t ask why we didn’t name it Roadworks! I have no idea. Anyway, This model is reading the data directly from our SDE (Spatial database) – if you can read that, it just says it is coming from a schema called Transport and that the feature class is called Streetworks. It is then passed through a number of transformers doing various things to the data and finally pushed into a custom transformer that adds certain fields that we have decided is necessary for all our datasets to hold. Such as the label ‘Nottingham City Council’ and our linked data URL on Ordnance Survey. It then writes to different formats. In this case it is our non spatial data so we’ve decided all non spatial data to be in XML, JSON and CSV.
• #17: This example is a little more messier. The only difference is that we are publishing this in spatial formats. So you’ll see a CSV that holds coordinate information, Shapefile, GeoJSon, Kml and an Esri Geodatabase feature class which is actually being used on our map view of the data.
• #18: In order to automate our FME Models. We use a Python script that is scheduled to run daily, weekly, monthly and quarterly. For example our street furniture model runs daily. It doesn’t give us the flexibility that FME Server would but due to limited resources and not being able to justify having FME server it is a solution that works for us. This automation using FME saves us a lot of time and without it we would be unable to keep Open Data Nottingham up-to-date.
• #19: Here’s a quick screenshot of our Python script. As you can see if you’re familiar with Python. We import any necessary modules at the top. Map a few drives that the data is stored in, as this script runs on a server that doesn’t have all the drive mappings by default. It then lists the FME workspaces to run as well as the log files and then runs through a loop to run each FME workspace. Finally it sends an email including the log file to tell me whether it has been successful or not. I can then investigate if something isn’t working as expected.
• #20: In conclusion! As I’ve said on the slide. FME is ESSENTIAL. Absolutely essential for allowing us to meet our Local Government Transparency Code. FME allows us to control and manipulate our data. It has allowed us to Open our data and deliver it to the public with minimal intervention due to the automation of our processes. And most importantly it’s saved us tremendous amounts of time and money!
• #21: Thank you very much for having me. -Smile!-