HDF and netCDF Data Support in ArcGIS
Download as PPTX, PDF3 likes5,133 views
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
HDF and netCDF Data Support in ArcGIS
- 1. HDF and netCDF Data Support in ArcGIS Nawajish Noman Jeff Donze HDF and HDF-EOS Workshop XV, April 17-19, 2012, Riverdale, MD
- 2. Outline • • • • • HDF and netCDF in ArcGIS Time in ArcGIS Performing Analysis Python Tools Future Directions
- 3. HDF and netCDF in ArcGIS
- 4. Scientific Data and Esri • Direct support - NetCDF and HDF • THREDDS/OPeNDAP – a framework for scientific data networking, integrated use by our customers • Examples using Esri technology • National Climate Data Center • National Weather Service • National Center for Atmospheric Research • U. S. Navy (NAVO) • Air Force Weather • Australian Navy • Australian Bur.of Met. • UK Met Office
- 5. HDF4 and HDF5 Support in ArcGIS
- 6. Raster Concepts • Raster Format (e.g. img, tif, etc.) – driver level support / storage format and layout / read/write of pixels and metadata • Raster Type (e.g. GeoEye-1) – implies Raster Format support and are format and sensor specific – intelligent use of metadata and other sensor specific parameters – defines georeferencing and well known processing chains • Raster Product (e.g. Panchromatic, Multispectral, Pansharpened ) – templates which make it easy to work with well defined end user products – multiple per sensor – e.g. Panchromatic, Multispectral, Pansharpened • Raster Product Definition (e.g. Natural Color, False Color) – defines “how you want your Mosaic Dataset to look” regardless of multiple source sensors and band combinations – uses metadata such as wavelength information to map bands
- 7. HDF Raster Support Raster Concept ArcGIS 10.1 Support Raster Format HDF4 • read: open a HDF subdataset as a Raster Dataset • write: APIs available but not exposed in UI HDF5 • read: open a HDF subdataset as a Raster Dataset • write: not supported at this time Raster Type HDF4, HDF5 • direct ingest of one or many HDF subdatasets into a Mosaic Dataset using the Raster Dataset Raster Type or the Table Raster Type * Esri interested in discussing other Raster Types Raster Product * Esri interested in discussing other Raster Products Raster Product Definition * Esri interested in discussing other Raster Product Definitions
- 8. HDF Raster Support • 10.1 Raster Format and Raster Type support implies… – ArcGIS Desktop • direct use as a Raster Dataset or Mosaic Dataset • use via conversion (i.e. convert to another format) • feature rich use in the applications – Visualization and Mapping – Geoprocessing – ArcGIS Server • publishing as dynamic image services • caching and publishing as tile services (i.e. basemaps) • OGC (WCS, WMS, WMTS)
- 9. Displaying MODIS LST Data HDFView ArcGIS
- 10. NetCDF Support in ArcGIS • ArcGIS reads/writes netCDF since version 9.2 • An array based data structure for storing multidimensional data. T • N-dimensional coordinates systems • X, Y, Z, time, and other dimensions • Variables – support for multiple variables • Temperature, humidity, pressure, salinity, etc • Geometry – implicit or explicit • Regular grid (implicit) • Irregular grid • Points Y Z X
- 11. Gridded Data Regular Grid Irregular Grid
- 12. Ingesting netCDF data in ArcGIS • NetCDF data is accessed as • Raster • Feature • Table • Direct read • Exports GIS data to netCDF
- 13. CF Convention Climate and Forecast (CF) Convention http://cf-pcmdi.llnl.gov/ Initially developed for • Climate and forecast data • Atmosphere, surface and ocean model-generated data • Also for observational datasets • The CF conventions generalize and extend the COARDS (Cooperative Ocean/Atmosphere Research Data Service) convention. • CF is now the most widely used conventions for geospatial netCDF data. It has the best coordinate system handling.
- 14. NetCDF and Coordinate Systems • Geographic Coordinate Systems (GCS) • X dimension units: degrees_east • Y dimension units: degrees_north • Projected Coordinate Systems (PCS) • X dimension standard_name: projection_x_coordinate • Y dimension standard_name: projection_y_coordinate • Variable has a grid_mapping attribute. • CF 1.6 conventions currently supports thirteen predefined coordinate systems (Appendix F: Grid Mappings) • Undefined • If not GCS or PCS • ArcGIS writes (and recognizes) PE String as a variable attribute.
- 15. NetCDF Tools Toolbox: Multidimension Tools • Make NetCDF Raster Layer • Make NetCDF Feature Layer • Make NetCDF Table View • Raster to NetCDF • Feature to NetCDF • Table to NetCDF • Select by Dimension
- 17. Changing Time Slice 143 342 341 441 131 231 331 431 121 Time = 1 241 221 321 211 311 232 332 433 223 323 423 213 313 413 432 122 222 322 422 112 212 312 412 421 111 333 442 132 141 233 113 242 443 123 142 343 133 Y 243 411 X Time
- 18. Using NetCDF Data Behaves the same as any layer or table • Display • Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers. • Graphing • Driven by the table just like any other chart. • Animation • Multidimensional data can be animated through a dimension (e.g. time, pressure, elevation) • Analysis Tools • A netCDF layer or table will work just like any other raster layer, feature layer, or table. (e.g. create buffers around netCDF points, reproject rasters, query tables, etc.)
- 19. Time in ArcGIS
- 20. Time is now built-in to ArcGIS • Simple Temporal Mapping • Unified experience for Time • Configure time properties on the layer • Use Time Slider to visualize temporal data • Share temporal visualization • • • • Time-enabled Map Services Export videos or images Generate temporal map books using ArcPy scripting Layer and map packages
- 23. Spatial and Temporal Analysis • Several hundreds analytical tools available for raster, features, and table • Temporal Modeling • Looping and iteration in ModelBuilder and Python
- 24. Generate Rainfall Statistics • Calculates specified statistics for all time steps • Outputs a raster catalog • Optionally outputs a netCDF file
- 25. Generate Rainfall Statistics Table • Calculates statistics for all time steps • Outputs a table • Optionally creates a graph
- 26. Python Tools
- 27. Community Developed Tools • Geoprocessing Resource Center http://resources.arcgis.com/geoprocessing/ • Marine Geospatial Ecology Tools (MGET) • Developed at Duke Univ. • Over 180 tools for import management, and analysis of marine data • Australian Navy tools (not publicly available)
- 28. Sample Script Tools • Python is used to build custom tools for specific tasks or datasets
- 29. NEXRAD Geoprocessing Tools • • • • • • Currently 6 geoprocessing script tools Designed to work with NEXRAD netCDF file Can be easily modified for other datasets Customized tools for various workflows Simplify repetitive work Automate GIS processes
- 30. Extract NEXRAD Rainfall To Points • Extracts the cell values for all time steps • Outputs a feature class
- 31. New NetCDF Tools (under development) • Download NetCDF File (OPeNDAP, WCS) • Clip • Extract By Variable • Extract By Dimension • Append By Dimension • Variable Statistics • Temporal Statistics
- 32. Download NetCDF File (WCS/OPeNDAP)
- 33. Dependencies on 3rd Party Utilities • netcdf4-python • This module can read and write files in both the new netCDF 4 and the old netCDF 3 format, and can create files that are readable by HDF5 clients. • Pydap • Pydap is a pure Python library implementing the Data Access Protocol, also known as DODS or OPeNDAP. OWSLib • OWSLib (OGC Web Service utility library) • Package for working with OGC map, feature, and coverage services. OWSLib provides a common API for accessing service metadata and wrappers for GetCapabilities, GetMap, and GetFeature requests.
- 35. HDF and Swath
- 36. Scientific Data Workshop and Future Initiatives… • Esri hosted a workshop in February 2012 • To understand the future need for scientific data support in ArcGIS • Ongoing efforts - require close collaboration with all of you • Some of the future initiatives are: • Continue to support netCDF classic and netCDF4 • • • • Provide better support for HDF5 Provide tool to consume data served using THREDDS/OPeNDAP Continue to support the evolving CF convention Support a strong developer experience for netCDF and HDF using Python • What else?
- 37. Things to Consider… • Embrace the Common Data Model (netCDF, HDF etc.) • Use Data and metadata standards (OGC, CF etc) • Provide “mechanism” so that we can access scientific data using a single set of APIs…. • and can expect data to be CF complainant • Make your data “spatial” (by specifying geographic or a projected coordinate system) • Clearly define workflow and requirements • Create sample tools where possible
- 38. Questions? Nawajish Noman Team Lead nnoman@esri.com Jeff Donze Esri Federal Business Development jdonze@esri.com
Editor's Notes
- #6: - Land Surface Temperature (LST)-The global monthly composite and average daytime and nighttime land-surface temperatures (LST) at grids with equal latitude and longitude bin sizes of 0.25 degree.- The LST data were generated from the daily MODIS LST product (MOD11B1) at5km spatial resolution.- MODIS (or Moderate Resolution Imaging Spectroradiometer) is a key instrument aboard the Terra (EOS AM) and Aqua (EOS PM) satellites.- http://modis.gsfc.nasa.gov/about/Data Source:Zhengming Wan (wan@icess.ucsb.edu) Institute for Computational Earth System Science University of California, Santa Barbara, CA 93106