Using FME to Convert TIGER Spatial Data From Oracle Spatial To ESRI Shapefiles
0 likes1,488 views
The document discusses the use of FME to convert TIGER spatial data from Oracle Spatial to ESRI shapefiles, emphasizing modernized approaches and automation in shapefile creation. It details considerations for maintaining data consistency and ease of maintenance, while outlining the processes for both public and internal shapefile production. The document also explains the use of Oracle wallet support for database credentials and provides contact information for further inquiries.
Using FME to Convert TIGER Spatial Data From Oracle Spatial To ESRI Shapefiles
- 1. Using FME to Convert TIGER Spatial Data From Oracle Spatial to ESRI Shapefiles Mei-Leng Freeman, Tim Puzycki, Jay E. Spurlin Spatial Products Software Branch Geography Division, Bureau of the Census
- 2. Introduction • Public products – TIGER/Line Shapefiles – Partnership Program Shapefiles • Internal products – Data Access and Dissemination Staff (DADS) shapefiles for use in the American Fact Finder websites (internal and external) – Live Individual Shapefile Request and Delivery System (LISRDS) – Quality Control (QC) shapefile for various projects 2
- 3. Shapefile Creation: Legacy Approach • Legacy TIGER – Utilized for 1990 and 2000 Decennial Censuses – In-house database format – Binary files – Partitioned by county • Legacy Shapefile Generation (shpGen) – Java shapefile classes – Perl driver scripts – Directly read from a TIGER partition – Legacy TIGER C API – Directly wrote to a shapefile – ASCII Job configuration files (user defined tags) 3
- 4. Modernized Approach • MAF/TIGER – The Legacy Master Address File (MAF) and Topologically Integrated Geographic Encoding and Reference (TIGER) datasets were combined. – Oracle Spatial and Oracle Topology Data Model v10.2 – Seamless, national data set • Current shpGen – Perl driver scripts – Pre-generated FME mapping files (.FME) – FME Engine handles reading from Oracle Spatial and writing to ESRI shapefiles 4
- 5. Initial Process • Considerations for automation – Automate everything possible. – Use COTS software, wherever possible. – Ensure consistency of data after conversion. – Provide ease of maintenance. – Make source code reusable. – Fairly stable source data structure. – Fairly stable target shapefile formats. 5
- 6. FME Mapping File Usage • Manually create FME mapping files (.FME) – Previous FME mapping files used as templates. – A standardized set of macros is used. • The FME mapping files are released as part of the software installation. • Automated production control system drives the process and provides any dynamic inputs. 6
- 7. Oracle Wallet Support • Stores database credentials: – Username – Password – Database • Avoids providing Username and Password to command line interface 7
- 8. Example FME Mapping File 8
- 9. Example FME Mapping File 9
- 10. Manually Created FME Mapping Files • TIGER/Line Shapefiles – Produced from “Benchmarks” (snapshots) of the MAF/TIGER Database. – Public TIGER/Line shapefiles are posted on the world wide web. • Partnership Shapefiles – MAF/TIGER Partnership Software (MTPS) • LISRDS Shapefiles – On-demand access to MAF/TIGER data – Data source may be “live” or a snapshot 10
- 11. Current process • A wider set of shapefile requirements – Mapping performance issues • Chained line representation of area boundaries • Un-duplication of lines in partitioned layers – Visual representation issues • Partial features • ‘Balance of’ features – Connect with Oracle Wallet 11
- 12. FME Mapping Files from FME Workbench • DADS/AFF Shapefiles – Both .FME and .FMW files are archived with the source code. – The .FME files are released to production, and used by FME Engine. – A similar procedure (though not yet as automated, and not making use of the geoprocessing tools) is used by another branch for the generalized and projected shapefiles. – Samples on the following slides: 12
- 13. 13
- 14. 14
- 15. 15
- 16. 16
- 17. 17
- 18. 18
- 19. 19
- 20. 20
- 21. Example: DADS/AFF Line Shapefiles State 05 State 13 State 09 State 02 21
- 22. Example: DADS/AFF Line Shapefiles State 05 State 13 State 09 State 02 22
- 23. 23
- 25. Workbench File Example: County Subdivisions 25
- 26. Example: Connect with Oracle Wallet 26
- 27. Thank You! • Questions? • For more information: – Jay E. Spurlin • jay.e.spurlin@census.gov – Bureau of the Census • 4600 Silver Hill Road, Stop 7400 • Washington, DC 20233 • Products made using FME Engine – TIGER/Line Shapefiles • http://www.census.gov/geo/www/tiger/index.html – American FactFinder (map base data) • http://www.factfinder.census.gov/home/saff/main.html?_lang=en