Cloud Optimized HDF5 for the ICESat-2 mission
0 likes63 views
The document discusses the migration of NASA and other agencies' data to the cloud, emphasizing the challenges of accessing HDF5 files due to their slow performance. It highlights the need for cloud-optimized HDF5, including strategies like consolidated metadata and reduced requests, to improve data accessibility and enable large-scale scientific research. The benefits of cloud optimizations are demonstrated, showing that performance enhancements can significantly reduce I/O requests.
Cloud Optimized HDF5 for the ICESat-2 mission
- 1. Cloud Optimized HDF5 for the ICESat-2 mission ESIP Summer meeting 2024 Luis López Research Software Engineer NSIDC Andrew Barrett Aleksandar Jelenak Lisa Kaser Jeff Lee Amy Steiker Credit: NASA's Goddard Space Flight Center
- 3. Important questions about our planet can now be answered by integrating years of data from different missions. Global Sea Ice Concentration Boreal Forest Biomass
- 4. The data coming from these missions is now available in the cloud! ** NASA and other agencies started to migrate their data to the cloud. **caveat: it’s by large in archival formats, HDF5 and NetCDF
- 5. Problem: Accessing HDF5 in the cloud is slow, how slow?
- 6. Improving performance of HDF5 in the cloud is key to enable science at scale. ● Data is becoming too large to work locally. ● I/O libraries are optimized for local and supercomputing workflows. ● HDF as a format was not designed for the cloud. The problem = Size + Tools + Format
- 7. Cloud-optimized HDF5? https://www.hdfgroup.org/2024/01/strategies-and-software-to-optimize-hdf5-netcdf-4-files-for-the-cloud/ ● Metadata is consolidated ● Custom caching buffer size ● Global API lock is still in place
- 8. ● Metadata is scattered through the file, each nested group makes this problem worse. ● By default, metadata is written to the file (and read from) on fixed blocks of 4kb. 1MB of metadata ~= 250 requests. ● Global API lock, those 250 reqs are sequential! Why HDF is not Performant in the Cloud
- 9. Paged Aggregation (data + metadata)
- 10. Metadata Blocks (user or dedicated page)
- 12. Trying Big Files from the ICESat-2 Mission Source: https://github.com/nsidc/earthaccess/discussions/251
- 13. Accidental Complexity NASA Policies file format library I/O driver data wrangling library AWS S3 ds = xr.open_dataset(“s3://nasa-data.hdf5”) (or ROS3)
- 14. It’s APIs All the Way Down
- 15. Cloud-Optimized HDF5 Works!* Code: https://gist.github.com/betolink/b545c364f80882c113b8cc27b763c729 Source: Andrew Barrett
- 16. Remote I/O Visualized https://github.com/ajelenak/ros3vfd-log-info ● Cloud optimizations to HDF5 reduces requests by an order of magnitude ● Data that’s not cloud optimized or is read with out-of-the-box parameters produces a lot of I/O
- 17. What could we do with CO-HDF? Xpublish Kerchunk SlideRule Happy researchers
- 18. Improving performance of HDF5 in the cloud is key to enable science at scale. Thanks!