HDF-EOS Development Current Status
0 likes302 views
HDF-EOS 5 is a new format based on HDF5 that is designed to resemble the existing HDF-EOS 2 format. It supports the same data structures as HDF-EOS 2 but uses HDF5 functionality like compression and chunking. A conversion tool allows data to be changed from HDF-EOS 2 to HDF-EOS 5 format. While HDF-EOS 2 will still be supported, HDF-EOS 5 will be used by new missions like Aura and provides more options for data storage and filtering.
HDF-EOS Development Current Status
- 1. HDF-EOS Development Current Status Larry Klein, Shen Zhao and Ray Milburn L-3 Communications EER Systems, Inc. December 5, 2002 Larry.Klein@L-3com.com, szhao@eos.east.hitc.com 1
- 2. HDF-EOS 2 • HDF4 based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by NCSA and NASA for the fore-seeable future. 2
- 3. Current Archive Holdings • GSFC DAAC: AIRS, MODIS L1, L2 Atmos./Ocean – 900 TB • EDC DAAC: Landsat/ASTER/MODIS Land – 500 TB • Langely DAAC: MISR – 300 TB • NSIDC DAAC: MODIS L2, L3 – 20 TB 3
- 4. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000. • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – – – – Supports same data structures Added prefix ‘HE5_’ to HDF-EOS 2 functions. Doesn’t preclude HDF5 functionality. Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG 4
- 5. HDF-EOS 2 -> HDF-EOS 5 •HDF-EOS 2 SWdefdatafield(swathID, fieldname, dimlist, numtype, merge) •HDF-EOS 5 HE5_SWdefdatafield(swathID, fieldname, dimlist, maxdimlist, numtype, merge, Maxdimlist*) *New HDF5 functionality passed through. This allows the user to set an upper limit to the size of the dataset. 5
- 6. HDF-EOS 5 • Provide a conversion tool: heconvert – HDF-EOS 2 -> HDF-EOS 5 – compliments HDF4 -> HDF5 conversion tool • HDF-EOS 5 will be used by EOS Aura – Standard format for profile data developed • One GLAS summary product in HDF-EOS 5 • Assume that HDF-EOS 2 producers will convert when PI’s determine that the time is right. 6
- 7. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL ARRAY SWATH GRID POINT Global (File) Attributes The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. 7
- 8. Swath Structure SwathName Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales Group CoLatitude Shaded Objects are implemented in a fixed way so the user doesn’t have direct access via the interface Attribute Data Set 8
- 9. HDF-EOS 5 Functionality • • • • • • • • • Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface Generalized Array Interface (Zonal Avg.) • Global (File), Group & Local Attributes • External Data Files • Subsetting • Unix/Linux/Windows • Threadsafe Version • FORTRAN, C, C++ 9
- 10. ECS support of HDF5 • ECS toolkit V 5.2.8 supports HDF4 and HDF5 - based applications – HDF-EOS 2.8 (hdf4.1r5) – HDF-EOS 5.3 (hdf5-1.4.3) • HDF5 and HDF4 must both be compiled – HDF4 users not affected – HDF5 users must use PGS_MET_SDstart() and PGS_MET_SDend() 10
- 11. ECS support of HDF5 • Both flavors of HDF-EOS (HDF) are part of the ECS baseline. • OS’s: Sun, SGI, IBM, HP, DEC, Windows NT/98, Linux • Compilers: FORTRAN 77/90, C, C++. • ECS will not crack HDF5 - based files for near future. (contents transparent to archive) • Size limit is 2 GBytes 11
- 12. Applications • HE5View (HDF-EOS 5 browser) • Java EOS Browser (HDF-EOS 2 and 5 access) • HDF-EOS processor: convert to GeoTIFF, subset, mosaic, subsample. In Data Pool interface (HDF4 based only) • heconvert (convert HDF-EOS 2 Grid/Point/Swath to HDF-EOS 5 equivalents) • Access libraries and applications at: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html 12
- 13. Issues • Only gzip (deflate) compression method implemented. • Metadata not updated for swath stitching in unlimited dimension. • File size limit in ECS - Is two Gbytes enough? • Currently in maintenance mode, but user requests are always welcome. 13