HDF5 OPeNDAP Handler Updates, and Performance Discussion
Download as PPTX, PDF0 likes152 views
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
- 1. SESIP-0722-KY HDF5 OPeNDAP Handler Updates, and Performance Discussion 2022 ESIP Summer Meeting This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001. This document does not contain technology or Technical Data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. Kent Yang Software Engineer/NASA EED-3 contractor myang6@hdfgroup.org
- 2. SESIP-0722-KY 2 • 2001: A prototype of HDF5 data handler – HDF5 to DAP***2: Default option • 2008: Handler in production – Climate and Forecast(CF) option: • Translate HDF5 metadata to follow CF • 2008-2018: Significant improvement – Still HDF5 to DAP2 HDF*5 OPeNDAP** Handler History * Hierarchical Data Format ** Open-source Project for a Network Data Access Protocol *** Data Access Protocol
- 3. SESIP-0722-KY 3 • Support DAP4 – CF option • Support 8-bit and 64-bit integer mapping – Default option • Support NetCDF* data model(group etc. ) • Documentation – A comprehensive user’s guide at github • https://github.com/OPENDAP/hyrax_guide/blob/master/handl ers/BES_Modules_The_HDF5_Handler.adoc HDF5 OPeNDAP Handler Update * Network Common Data Form
- 4. SESIP-0722-KY 4 • Output NetCDF file via the handler – Sometimes it is very slow HDF5 Handler Performance Study HDF5 File Hyrax Core HDF5 handler File netCDF NetCDF File
- 5. SESIP-0722-KY 5 • Because HDF5 variables are compressed. HDF5 Handler Performance Study
- 6. SESIP-0722-KY 6 HDF5 Handler Performance Study • How compressed variables are processed – HDF5 handler: Decompress via H5Dread – File NetCDF: Compress via H5write HDF5 File HDF5 handler File NetCDF NetCDF File Decompress Compress Hyrax Core
- 7. SESIP-0722-KY 7 HDF5 Handler Performance Study • Compression/decompression is costly • Solution – Passing through the compressed data HDF5 File HDF5 handler File NetCDF NetCDF File Decompress Compress Hyrax Core Pass through the data Pass through the data
- 8. SESIP-0722-KY 8 HDF5 Handler Performance Study HDF5 File HDF5 handler File NetCDF NetCDF File Hyrax Core Pass through the data Pass through the data • Is this possible? • A proof-of-concept Study
- 9. SESIP-0722-KY 9 HDF5 Handler Performance Study • A proof-of-concept study – Use HDF5 direct chunk IO* API**s • Packages that need to be updated – HDF5 handler • Read the passing-through compressed data – DAP library • Pass through the variable storage information – NetCDF-4 • Write the passing-through compressed data * Input Output ** Application Programming Interface
- 10. SESIP-0722-KY 10 HDF5 Handler Performance Study • Testing Files Used – GHRSST* and MERRA-2** data • Repack the data to one chunk per variable • Test Approach – Only Hyrax Back-End Server(BES) – besstandalone program on a Linux server – Measure the wall clock time to output a NetCDF-4 file GHRSST: Group for High Resolution Sea Surface Temperature MERRA: Modern-Era Retrospective analysis for Research and Applications
- 11. SESIP-0722-KY 11 HDF5 Handler Performance Study • Testing Files – GHRSST • File size: 237 MB • About 20 variables • 5392x3200 8-bit or 16-bit integer – MERRA-2 • File size: 489 MB • About 50 variables • 24x361x576 32-bit floating-point
- 12. SESIP-0722-KY 12 Performance Study Results • Performance improved ~17 and ~30 times compared to the standard way Wall Clock Time(Seconds) MERRA2 GHRSST Standard Way (Decompress and compress the data) 55 26 Pass through the compressed data 1.8 1.5 Speed up ~ 30 ~17 • Credit to the HDF5 library.
- 13. SESIP-0722-KY 13 This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001.