HDF-EOS Tools
Download as PPT, PDF1 like692 views
The document describes HDF-EOS tools developed to help users work with Earth Observing System (EOS) data stored in HDF-EOS format. It outlines libraries and tools created, including the HDF-EOS to GeoTIFF Converter (HEG) tool, which can convert HDF-EOS files to GeoTIFF and perform subsetting, stitching, and reprojection. It also describes the HDF-EOS plugin for HDFView, which allows browsing of HDF-EOS data and objects. The tools were created to make HDF-EOS data more accessible to users and reduce the need to learn HDF/HDF-EOS libraries.
HDF-EOS Tools
- 1. HDF-EOS Tools Abe Taaheri, Raytheon IIS ESIP Meeting Chapel Hill, NC July 9, 2013 Page 1
- 3. Overview • The HDF has been selected by the EOSDIS Project as the format of choice for standard product distribution of EOS data • Under NASA contracts (ECS, EMD, EEB, EED) we developed HDF-EOS, and performed other development and maintenance activities Page 3
- 4. Overview – Libraries HDF-EOS 2 & 5 SDP & MTD Toolkits – Tools HDF-EOS2 to GeoTIFF Converter (HEG) HDF-EOS plug-in for THG HDFView HDF-EOS2 to HDF-EOS5 conversion – User support Helpdesk for Toolkit/HDF-EOS/HEG Page 4
- 5. Tools Why we need some HDF-EOS tools ? Page 5
- 6. Why HDF? • HDF was selected for EOS data since – Many of the HDF defined datatypes map well to EOS datatypes Raster images, multi-dimensional arrays, and text blocks – HDF provides efficient data management Storage & performance Page 6
- 7. HDF-EOS Datatypes • Some other EOS datatypes do not map directly to HDF datatypes, particularly ̶ geolocated datatypes Therefore, some additions to traditional HDF are required to fully support these datatypes. Page 7
- 8. HDF-EOS Datatypes To bridge the gap between the needs of EOS data products and the capabilities of HDF, 4 new EOS specific datatypes were introduced: Point, Swath, and Grid Zonal Average (hdf-eos5) Page 8
- 9. HDF-EOS Datatypes • Point : Unorginzed data (spatial or temporal) that has associated geolocation information • Swath: Time-ordered data such as satellite swaths (time-ordered series of scanlines), or profilers (time-ordered series of profiles). • Grid: Data stored in a rectilinear array based on a well defined and explicitly supported projection. • ZA: Swath like datatype without geolocation mapping Page 9
- 10. HDF4 File Contents – User View Objects & Relationships Object Data User Metadata Page 10
- 11. HDF4 File Contents – Format View variable name = variable_name rank type storagetype 1 Vgroup name = variable_name class = Var0.0 1 1 Object Data 1 1 1 0...1 1 SD SDD 1 0...1 data 0…* byte order, chunked storage, compression, … attribute name = attribute_name HDF4 Mapping Project Update - Ruth Aydt 1 1 0...1 NT 1 1 1 1 1 1 NDG 0…* Vdata name = attribute_name class = Attr0.0 Page 11
- 12. HDF/HDF-EOS are Complicated! • HDF is not efficient for Users since they need to learn - how to use HDF & HDF-EOS libraries - how to process the data correctly • Licenses for commercial tools are expensive The HDF Group & Raytheon have developed some tools to help users Page 12
- 13. HDF & HDF-EOS Tools For most users HDF readers or NetCDF/CF conversion applications are enough - hdp, h5dump, eos2dump(for lat/lon), HDFView - h4tonccf, h4cf, OpeNDAP Handlers But some need more processing and tools appropriate for EOS data • HDF-EOS to GeoTIFF Conversion Tool (HEG) • HDF-EOS plug-in for HDFView Page 13
- 14. Page 14
- 15. HEG Name says HEG is: A utility to convert EOSDIS data from HDF-EOS2 to GeoTIFF format. But in practice: It can do more than just conversion to GeoTIFF Page 15
- 16. HEG Functionality • Conversion – HDF-EOS (Swath or Grid ) to a single-band or multi-band GeoTIFF’s – HDF-EOS Swath to HDF-EOS Grid --> GeoTIFF OR – ASTER Swath is written directly into GeoTIFF – HDF-EOS Swath or Grid to generic Binary (with metadata file) Page 16
- 17. HEG Functionality • Subsetting – spatial: using Swath/Grid corner lat/lon or pixel numbers – field: Grid, Swath, and Field selection – band: Selection of data layers in 3-D and 4-D datasets ( 3-D to 6-D MISR SOM datasets ) Page 17
- 18. HEG Functionality • Stitching – (mosaicing) + subsetting + Reprojecting • Reprojection ̶ To: UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers, CEA • Subsampling ̶ Subsample stacks Page 18
- 19. HEG Functionality • Running from Command Line (batch jobs) or • with Java-based GUI • Metadata preservation/creation. Page 19
- 22. HEG Conversion GUI Bilinear Nearest Neighbor Cubic Convolution Geographic Polar Stereographic UTM Mercator Transverse Mercator Lambert Azimuthal Lambert Conformal Conic Sinusoidal Albers State Plane Cylindrical Eqyual-Area No-Reprojection Lat-Lon Rows-Columns Page 22
- 23. HEG HEG Parameter File Used for running HEG on command line. Page 23
- 24. Status log for HEG run Page 24
- 25. HEG Data/Platform Support • Currently supports MODIS, MISR, ASTER, AIRS, AMSR-E products on TERRA and AQUA – more than 210 products • Operable on Sun, Windows, Linux, MAC Page 25
- 26. HEG Versions Versions Available: Stand-alone: A downloadable desktop version with a User Interface ̶ Portable, written in Java. ̶ Not dependent on COTS (eg. IDL). Page 26
- 27. HEG Versions Integrated into ECS Data Pool: Access through NASA archive online storage ̶ Reduces the transfer time of HDF-EOS data sets (if subset is requested) ̶ Provides the end-user with the exact file required by their application Page 27
- 28. Examples (MISR) Mt. Etna Eruption: MISR L1B Stitched Blocks Page 28
- 29. Examples (MODIS) 2004 Hurricane Francis, MODIS MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5 Page 29
- 30. Examples (ASTER) AST_L1B, 2003 California Fires Geo. projection, Multi-Band GeoTIFF (3 bands): Visible Near Infrared (VNIR) ….. and Subsetted Two Swaths Stitched ….. Page 30
- 31. Examples ( AMSR_E_L2_Ocean) swath Swath Conversion to Geographic Grid/HDF-EOS output Page 31
- 33. Page 33
- 34. HDFView A Java-based browser from The HDF Group for HDF4 and HDF5 based data. Page 34
- 35. HDFView with HDF-EOS Plug-in Page 35
- 36. HDFView with HDF-EOS Plug-in –Extends HDFView to browse HDF-EOS data in both versions (2 & 5). –Is a plug-in module does not change HDFView functionality –Identical look and feel when displaying HDF and HDF-EOS objects –Access to complete breakdown of HDF-EOS objects. e.g. grid info, projection info, dims, datafields, and attributes Page 36
- 37. HDFView Plug-in What else HDF-EOS plug-in adds to HDFView? Page 37
- 38. Swath, Grid, and Point Objects Plug-in file treeview Swath/Grid Lat/Lon display 38 Page 38
- 39. HE2 to HE5 Conversion One click to convert HE2 to HE5 file format 39 Page 39
- 40. AMSR-E Swath Getting Image of selected pixels in a table Getting lat/lon for a selected pixel in a table 40 Page 40
- 41. MODIS (HE2) and OMI (HE5) Grids Drawing Coastlines & International Borders on HE2 & HE5 Grids 41 Page 41
- 42. Future work • Coastlines & International borders for HE2 & HE5 swath • Adding single-band/multi-band GeoTIFF writer • Adding GeoTIFF reader with geolocating capability Page 42
- 43. Availability • Download, Documentation, Help: HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/ HDFView/HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/ HEG/HEGHome.html Page 43
- 44. Help, Comments • Questions, Comments, Requests: Abe_Taaheri@raytheon.com PGSTLKIT@raytheon.com http://hdfeos.org/forums/index.php Page 44
Editor's Notes
- #11: Look a bit more closely at the types of information that are in an HDF4 file… all of these are important.The object data typically takes up most of the space in an HDF4 file and is the most difficult to represent in Text format.
- #12: If we look “under the hood” things are more complicated than they appear from the HDFview display…. The HDF4 library typically hides the complexity. The car’s engine is kind of like the object data in the HDF4 file… challenging to represent and explain. Bits of data for a single object can be spread throughout the file, compressed, etc. Can we represent this in a way that something other than the HDF4 library can find the right bits & reconstruct the data?