HDF-EOS Development: Current Status and Tools
1 like318 views
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
HDF-EOS Development: Current Status and Tools
- 1. HDF-EOS Development Current Status and Schedule Larry Klein, Shen Zhao, Abe Taaheri and Ray Milburn L-3 Communications Government Services, Inc. September 24, 2003 1
- 2. What is HDF-EOS ? • HDF (Hierarchical Data Format) is a disk-based format chosen by NASA as the standard for the EOS program • HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures – By providing a single interface to data structures common to earth science – By providing a container for EOS inventory, archive and product specific metadata – By establishing a consistent relationship between geolocation and science data. 2
- 3. HDF-EOS Grid Structure • Usage - Data which is organized by regular geographic spacing, specified by projection parameters. • Structure – Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) – Geolocation information contained in projection formula, coupled by structural metadata. – Any number of Grid structures per file allowed. 3
- 4. HDF-EOS Swath Structure • Data which is organized by time, or other track parameter. • Spacing can be irregular. • Structure – Geolocation information stored explicitly in Geolocation Field (2-D array) – Data stored in 2-D or 3-D arrays – Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural metadata 4
- 6. HDF-EOS Point Structure • Data is specified temporally and/or spatially, but with no particular organization • Structure – Tables used to store science data at a particular Lat/Long/Height – Up to eight levels of data allowed. Structural metadata specifies relationship between levels. 6
- 7. Zonal Average Structure • Generalized array structure with no geolocation linkage • Data can be organized by time or track parameter • Data spacing can be irregular • Structure – Data stored in multidimensional arrays – Time stored in 1-D or 2-D array 7
- 8. HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • Current archive holdings: ~1.5 PB, 30M granules • HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future. 8
- 9. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different user interface. – First released in 2000. – Format to be used by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Supports same data structures – Added prefix ‘HE5_’ to HDF-EOS 2 functions. – Doesn’t preclude HDF5 functionality. 9
- 10. HDF-EOS Functionality • Basic File I/O • Fill Values • Compression (gzip, szip (Oct. 03)) • Chunking/Tiling • Swath Interface • Grid Interface • Point Interface • Profile Interface • Global (File), Group & Local Attributes • External Data Files • Subsetting • Threadsafe Version • Generalized Array Interface (Zonal Avg. (HDF5)) 10
- 11. HDF-EOS 5 • HDF-EOS 5 is used by EOS Aura producers and users. Launch Jan. 04. • Assume that HDF-EOS 2 users will convert when PI’s determine that the time is right. 11
- 12. Development Since December, 02 • Added the zonal average interface ZAapi.c. • Added two new functions HE5_SWinqdfldalias() and HE5_SWinqgfldalias() to retrieve information about data/geolocation fields and aliases in Swath. • Added a new function HE5_EHinqglbdatatype() to retrieve the data type information from a Global File attribute. • Szip compression (next release). • Next release - mid-Oct. linked with HDF5-1.6.1. 12
- 13. ECS Support of HDF-EOS • Both flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris, Irix, HP, AIX, DEC, Windows NT/98, Linux (64-bit soon), Mac OS X (soon) • Compilers: FORTRAN 77/90, C, C++. • Size limit per granule is 2 GBytes 13
- 14. Applications • • • • HE5View (HDF5 - based browser) EOSView (HDF4 - based browser) Java EOS Browser (HDF-EOS 2 and 5 access) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) – Compliments HDF4 -> HDF5 conversion tool – Not tested on all products 14
- 15. HDFView • HDFView: A Java-based browse tool from NCSA for HDF4 and HDF5 – based data. • We are currently extending HDFView to browse HDF-EOS data in both versions. – One reason: EOSView is becoming obsolete • The extension is a plug-in module and will not change functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects will remain. 15
- 16. HDFView (Cont.) • The look and feel of HDFView will be identical when displaying HDF and HDF-EOS objects. • Users will access a complete breakdown of HDFEOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes will be displayed. • A beta release is scheduled for end of Oct-03. 16
- 17. HDF-EOS to GeoTIFF Converter(HEG) • A utility that converts EOSDIS data from HDFEOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: – A downloadable desktop version – Access through NASA archive online storage (Data Pools) • Details of internal file structures are transparent to users. 17
- 18. HEG Functionality • Performs conversion to a single-band or multi-band GeoTIFF’s • HDF-EOS Swath to HDF-EOS Grid • HDF-EOS Swath or Grid to GeoTIFF • HDF-EOS Swath or Grid to generic binary (with metadata file) • Subsetting (spatial, temporal, parameter) • Stitching (mosaicing) 18
- 19. HEG Functionality • Reprojection (USGS, MODIS Integerized Sinusoidal, EASEGrid) • Subsampling (Subsample stacks) • Boundary smoothing • Metadata preservation/creation. • Currently supports MODIS, MISR, and ASTER products (>70 products) (AIRS proposed) • Allows file selection from users local storage 19
- 20. HEG (cont.) • • • • Java-based GUI Command line interface Operable on Sun, SGI, Windows, Linux Integrated into ECS Data Pool – Reduces the transfer time of HDF-EOS data sets (if subset is requested) – Provides the end-user with the exact file required by their application • User Interface – Portable, written in Java. – Initially operable on Sun, SGI, Linux. – Not dependent on COTS (eg. IDL). 20
- 21. OpenGIS Interface • Ionic’s Open GIS Consortium (OGC) Open Web Service (OWS), Web Coverage Service (WCS), Version 3.1. • Uses HEG • Will expose EOS products to tools on global Open GIS network. • Integration into the ECS Data Pool is planned for 2004 21
- 22. Availability • Access to libraries and applications: http://newsroom.gsfc.nasa.gov/sdptoolkit/to olkit.html • Email – – – – larry.klein@l-3com.com ray.milburn@l-3com.com shen.zhao@l-3com.com abe.taaheri@l-3com.com 22
- 23. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL ARRAY SWATH GRID POINT Global (File) Attributes . 23
- 24. Swath Structure SwathName Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales Group CoLatitude Shaded Objects are implemented in a fixed way so the user doesn’t have direct access via the interface Attribute Data Set 24