Plans for Enhanced NetCDF-4 Interface to HDF5 Data
0 likes413 views
The document outlines plans for an enhanced NetCDF-4 interface to HDF5 data, aiming to combine the simplicity and widespread use of NetCDF with the performance and generality of HDF5. Key features of NetCDF-4 include support for large files, parallel I/O, and new data types, while maintaining compatibility with existing NetCDF-3 applications. The project is funded by NASA's AIST program and acknowledges support from the National Science Foundation.
Plans for Enhanced NetCDF-4 Interface to HDF5 Data
- 1. Plans for an Enhanced NetCDF-4 Interface to HDF5 Data Russ Rew, UCAR Unidata Program Mike Folk, NCSA/UIUC Ed Hartnett, UCAR Unidata Program Robert E. McGrath, NCSA/UIUC
- 2. What is NetCDF? • Similarities with HDF5: A scientific data model A data access library and set of interfaces A format for storing and accessing data • Differences from HDF5: Simpler, more widely used (currently) Less powerful, less performance-oriented • Current version: netCDF-3 is supported by UCAR Unidata Program
- 3. NetCDF-4/HDF5 Goals • Combine desirable characteristics of netCDF and HDF5, while taking advantage of their separate strengths: Widespread use and simplicity of netCDF Generality and performance of HDF5 • Preserve format and API compatibility for netCDF users • Demonstrate benefits of combination in advanced Earth science modeling efforts
- 4. NetCDF-4 Features Enabled by HDF5 • • • • • • • Large file support Parallel I/O Multiple dynamic dimensions Packed data, compression New data types Dynamic schema modifications Other possibilities: groups, Unicode, userdefined types, better coordinate support, …
- 5. Backward Compatibility • Two kinds of compatibility: API: Extension of netCDF-3 API Support existing netCDF-3 applications by recompiling/relinking Format: Access to existing netCDF-3 data Access to HDF5 files written through netCDF-4 interface • Leverage C interface for other interfaces to Fortran90, C++, Perl, Ruby, … • Java interface must be developed separately
- 6. NetCDF-4 Architecture netCDF-3 netCDFnetCDF-3 applications applications netCDF netCDF files files netCDF-4 HDF5 files HDF5 files netCDF-4 netCDFnetCDF-4 applications applications HDF5 HDF5 applications applications netCDF-3 Interface netCDF-4 Library HDF5 Library • Supports access to netCDF files and HDF5 files created through netCDF-4 interface
- 7. NCSA Contributions • Overall design • Dimension scales To support shared dimensions Addition to HDF5 data model • Parallel I/O Using Argonne/Northwestern work Challenging performance goal
- 8. Acknowledgments and URL • The netCDF-4/HDF5 project is funded by the NASA AIST program • Unidata is sponsored by the National Science Foundation • URL: www.unidata.ucar.edu/packages/netcdf-4