How to Manage Open Police Data - Tips for Data QA/QC and Automation
2 likes1,956 views
The document discusses managing open police data, emphasizing the importance of transparency, ethical data sharing, and accurate information. It details the experiences and tools used by the City of Auburn, including automation with FME for data transformation and integration into GIS for improved public reporting and analysis. Additionally, it reviews other cities' efforts toward enhancing their open data platforms for better accessibility and usability.
How to Manage Open Police Data - Tips for Data QA/QC and Automation
- 1. How to Manage Open Police Data – Tips for Data QA/QC and Automation Alice Pence, City of Auburn Robert Schultz, RCMP in BC Cat Gracey, Safe Software Dale Lutz, Safe Software
- 2. Who are we? Connect Your Applications | Transform Your Data | Automate Your Workflows
- 3. Police Data Initiative 53 U.S. jurisdictions. 41 million people. 90 datasets. And growing! www.publicsafetydataportal.org
- 4. Why Should Police Data be Shared? ● Increase transparency and build trust ● Help provide improved public services ● Fuel more intelligent policing decisions The Ethics: ● Need to protect victim identity ● Only share when it’s lawful to do so ● Must be accurate, free of errors
- 5. Dallas Police Department Experience (by the numbers) 12
- 6. Dallas Police Department Experience (by the numbers) 45
- 7. Dallas Police Department Experience (by the numbers) 67
- 8. Alice Pence City of Auburn
- 9. Automating Crime Data to Import into GIS Alice Pence GIS/Database Specialist City of Auburn, WA
- 10. Quick Bio • BS in Geography from University of Idaho w/Certificate in GIS • In GIS field for 3+ years – been in Auburn, WA for over 1 year Alice Pence
- 11. Our Crimes Database • Stored in a Proprietary Linux system • Raw data with only addresses • Can’t directly dump into GIS- needs to have data transformed for public/internal viewing, QA/QC, a spatial component to view on maps and applications, attributes renamed…
- 13. Our Mission How to view all this data spatially for analysis and reporting? Our first idea…
- 14. A Python Script The Good • It worked. Can add new offenses, change and update as needed. • Ran in windows scheduler to automate. The Bad • Long, cumbersome script. • Needed multiple databases (.gbd → SDE) to clean and move data. The Ugly • Can’t easily change connection to Crimes Database, no direct connection to source (connected through ArcCatalog -then built through Model Builder- then exported out as a Python Script). • No custom alerts or in depth logging capabilities. • Not user friendly for non-python users.
- 15. An easier solution? Our next idea…
- 16. FME!!
- 17. First Challenge Python FME This continues on for a few more pages… How to rename and group 4 letter offense codes
- 18. Attribute Mapper Transformer • As an example: RKBK Crime Type= Felony Crimes Against Person Crime Name= Robbery 1 2 3 *double bonus: I was able to automatically import codes by a CSV file, AND…FME automatically read in all the source values for me
- 19. The Rest of the Process
- 20. Into the GIS Environment FME → SDE → MXD → Web Map Service → Online Application = Reporting and Analysis
- 21. Wait- there’s more… Automating the workbench and logging too!
- 22. FME Server Set Up Email notifications when Workbench succeed or failed
- 23. Logging Capabilities (FME Jobs)
- 24. Main point: FME rocks
- 25. Comparison Python • Cumbersome/long script • Not very easy to customize alerts or logs • Hard to edit- other users might not know the python language • No friendly user interface FME • Easy to update codes and field changes • Other users can easily edit- even w/minimal knowledge of FME • Share directly to FME Server for logs and alerts • Friendly user interface for FME Server and easy to set up
- 26. Our Next Great Idea… Socrata Open Data Portal
- 27. Open Data Portal: Crimes Dataset Updated nightly with new data, thanks to FME Desktop and FME Server
- 28. FME Workbench
- 29. API Download and Export Tools Custom Data Lenses (aka Graphs and Maps
- 30. Thank you! Alice Pence GIS/Database Specialist City of Auburn, WA apence@auburnwa.gov Links to sites discussed in slides: Socrata Open Data Portal: https://data.auburnwa.gov/ Auburn Crimes Map: https://maps.auburnwa.gov/html5viewer/Index.html? viewer=crimes
- 31. Q&AChat your questions into the GoToWebinar control panel
- 33. Property Crime
- 34. Spatially Anonymized Property Crime
- 35. Property Crime – High Density
- 36. Property Crime – Low Density
- 37. Open Data in San Francisco The new ETL job platform.
- 38. Background. ● City’s official open data portal is SF OpenData. The Vision. ● Empower use of City data. “Our vision is that the City’s data is understood, documented, and high quality.”
- 39. The Project: Increase number and timeliness of datasets ● Few workspace authors ● Safe place to try out workspaces The Results. ● Improved ETL job platform, design and architecture ● 100-500 ETL jobs ● Simple ETL Jobs ● Use best-practices View the original presentation by Samuel Valdez and Janine Heiser (Presented at the FME World Tour 2016)
- 40. Open Data in NYC GIS for public use.
- 41. Background. ● The Citywide Street Centerline Database (CSCL) is the authoritative source for location data used by New York City's public safety agencies (NYPD, FDNY) for 911 call-taking and dispatch. The Vision. Make CSCL data easily accessible for the day-to-day GIS user. View the original presentation by Chris Rado (Presented at the FME World Tour 2016)
- 42. The Results. New version of CSCL created that is better suited for general GIS tasks and made it available to all city agencies, to the public via NYC Open Data and to Batman.
- 43. Q&AChat your questions into the GoToWebinar control panel
- 44. Thank you! Live chat us at www.safe.com Free Resources: FME Desktop Training (June 7-8) FME Server Training (June 14-15) Blog post: Guide to Open Data: Using it, Sharing it, and Creating a Portal Open Data Webinar Open Data eBook