HDF Update
Download as PPT, PDF1 like309 views
The document provides an update on the HDF software projects. It discusses recent releases of HDF4, HDF5, and HDF Java products. It highlights new features, platforms supported, and organizations contributing to development. Upcoming work includes improvements to parallel I/O, data indexing and viewing tools, and harmonization with netCDF and OPeNDAP formats.
HDF Update
- 1. HDF Update Mike Folk National Center for Supercomputing Applications HDF and HDF-EOS Workshop VIII October 27, 2004 -1- HDF
- 2. Topics • HDF Team and Supporters • HDF software update • Other Activities of Interest -2- HDF
- 3. The HDF Team Xuan Bai Frank Baker Peter Cao Vailin Choi Mike Folk Barbara Jones Quincey Koziol James Laird Raymond Lu John Mainzer Robert McGrath Pedro Nunes Elena Pourmal Binh-minh Ribler Eric Shapiro Rishi Sinha Kent Yang And all those wonderful folks out there who contribute ideas, requests, bug reports, code, and support. -3- HDF
- 4. Organization HDF Project Support, doc, doc, QA, QA, maintenance maintenance Basic library development Tools and Tools and Java Parallel I/O, Grid, Grid, big machines • Staff breakdown – – – – – User support, documentation QA, maintenance, testing Software development System administration Management • See Thursday tutorial on HDF Software Process -4- HDF
- 5. Who is supporting HDF? • Organizations and communities with institutional and financial commitment to HDF – NCSA, NASA, State of IL, DOE, Boeing • Agencies supporting R&D – NCSA, NASA, NARA, DOE, NSF, ONR • Collaborators who make in-kind contributions – Cactus, PyTables, NeXUS, CGNS, many others -5- HDF
- 7. HDF software milestones in FY 2004 HDF 4.2r0 HDF5 1.6.2 HDF5 Java 2.0 HDF5 High Level Flexible parallel HDF5 (Alpha) HDF5 1.6.3 Dec Jan Feb Mar Apr May Jun 2003 2004 -7- Jul Aug Sep Oct HDF
- 8. HDF4.2 Release 0 – Dec. 2003 • Bug fixes • New features • Support for new platforms and compilers -8- HDF
- 9. HDF4.2r0 New Features • Tools (per DAAC and Instrument Team requests) – hdfimport – converts float/integer data to SDS/raster • Replaces fp2hdf – Hdiff – compares two HDF4 files • Revision of earlier hdfdiff tool – Hrepack – makes a copy of an HDF4 file • optionally rewrite objects with compression, chunking, etc. – h4cc, h4fc, h4redeploy • Helper scripts to facilitate compilation and installation - 10 - HDF
- 10. HDF4.2r0 New Features • Szip compression – – – – – Fast compression method Available on all platforms except Crays NCSA distributes Szip source and binaries HDF Library binaries come with SZIP enabled SZIP Documentation available from http://hdf.ncsa.uiuc.edu/SZIP - 11 - HDF
- 11. HDF4.2r0 New Configuration • Addressing key needs – – – – Porting to new platforms New versions of JPEG and ZLIB libraries Optional SZIP compression Many features were hard coded, but could be done at configuration time - 12 - HDF
- 12. HDF4.2r0 New Compilers and Platforms • New compilers – Intel C and Fortran – Portland Group Compilers (C only for now) • New OS – – – – – – Mac OSX RedHat 8/9 AIX 5.1 64-bit OSF1 Linux 64 (SuSE and RH8) (JPL machines) Altix (Aura Team) - 13 - HDF
- 13. HDF5 1.6.2 – Feb. 2004 • New functions – better user control over open/close objects • Bug fixes • Parallel improvements – h5pcc, h5pfc helper scripts for parallel compiles – Configure improvements – Improved parallel performance • Speed improvements of data conversion routines • Some SZIP improvements - 14 - HDF
- 14. HDF5 1.6.2 • Support for new compilers and platforms – – – – IBM Fortran on MacOS X Support for gcc 3.3.4 Linux 64 (SuSE and RH) at JPL Altix (Aura team) including parallel C and Fortran Libraries – Investigated SX-6 (NEC) port - 15 - HDF
- 15. HDF5 1.6.3 – Oct. 2004 • Windows – Improvements to the build, test, and installation • New API routines – H5Fget_filesize. Returns size of opened file. – New: H5Fget_name. Returns name of file by object ID – Some F90 and C++ routines added - 16 - HDF
- 16. HDF5 1.6.3 • Utilities – H5repack utility (new) • Regenerates an HDF5 file from another HDF5 file, • Optionally applies filters, chunking to new file – H5dump utility improvements • Print new info, such as dataset filters, storage layout, fill value info - 17 - HDF
- 17. Szip in HDF5 1.6.3 • HDF5 can now include SZIP compression with or without Szip's encoder – Required to create SZIP compressed files – Not required to read SZIP compressed files • Info on Szip and Szip licensing: • http://hdf.ncsa.uiuc.edu/doc_resource/SZIP/ - 18 - HDF
- 18. HDF5 1.6.3 New platforms & compilers • PGI Fortran for Linux64 (x86-64) • Absoft F95 for Linux 2.4 -32 bit • IBM XL Fortran and Absoft F95 for Mac OS X - 19 - HDF
- 19. HDF Java Products 2.0 – March 2004 • Tested with HDF5-1.6.2 • Platforms – – – – – – – Windows (98/NT/2000/XP) Solaris Linux AIX IRIX 6.5 Mac OSX OSF1 • http://hdf.ncsa.uiuc.edu/hdf-java-html/ - 20 - HDF
- 20. Modular HDFView Modular HDFView – improved HDFView where I/O and GUI components are replaceable modules. • Replaceable modules: – File I/O (file/data format) – Tree view (show file structure) – Table view (spreadsheet-like) – Text view (view/edit text dataset) – Image view (view/process image) – Palette view (view/change palette) – Metadata (attribute) view • Application (HDFView) Interfaces I/O, TreeView, TableView, etc Default User Implementation Implementation http://hdf.ncsa.uiuc.edu/hdf-java-html/hdfview/ - 21 - HDF
- 21. HDFView Web Browser Plug-in • Goal: Click-and-view HDF files remotely and locally from popular web browsers. • See poster. - 22 - HDF
- 22. Parallel HDF5 in 2004 • A few performance improvements • MPICH/MPE instrumentation feature added – performance analysis tools for their MPI programs • “Flexible parallel HDF5” programming model – More flexible model for parallel HDF5 – Other options currently under investigation - 23 - HDF
- 23. Parallel HDF5 developments • New parallel platforms supported – – – – – Solaris 2.8 (32 & 64 bits) OSF 5.1 Cray T3E, SV1, T90 HPUX 11.0 FreeBSD - 24 - HDF
- 24. Other Activities of Interest - 28 - HDF
- 25. DOE/ASCI* “ASCI provides the integrating simulation and modeling capabilities and technologies needed …for future design assessment and certification of nuclear weapons and their components” • Massively parallel computing and I/O • Complex data models and big data • HDF5 a standard format for ASCI apps * “Advanced Simulation and Computing Program” - 29 - HDF
- 26. Boeing HDF5 for real-time flight test data • • • • • • • • Needed for flight test data systems Must handle raw, real-time data Implemented API to read/write data Based on HDF5 “table” API Challenge: Variable length data Possible Boeing-wide standard Potential applications to many domains See poster - 30 - HDF
- 27. NCASSR*: Indexing & viewing tables • Opportunities arising from Boeing work – Make test-data features widely available – Common data model and API for tabular data in HDF5 – Indexing for post-processing – Viewing capabilities • Tasks – – – – Identify apps to study and gather requirements Develop data model and API for tabular data Include general purpose indexing structures and API Implement prototype API and viewer * National Center for Advanced Secure Systems Research - 31 - HDF
- 28. National Archives and Records Administration (NARA) • Investigate HDF5 as format for records archiving • Focus on geospatial data – Images (e.g. elevation models, aerial photography) – Features (e.g. boundaries, roads, rivers) • Results so far – HDF5 data model handles all data types – Feature (vector) data present access and size challenges – Work is leading to good performance lessons • See poster about study of vector data - 32 - HDF
- 29. SciDAC/PMODEL Arithmetic Data Transform • Apply algebraic operations to dataset during read/write. • Initial goal: – transform individual elements (e.g., x * 1.8 + 32). – During reads, applies to result in memory. During writes, data in the file changed. • Implemented in HDF5 v1.7, to be released in v1.8 • Future – Transformations on attributes or multiple datasets (e.g. (A + B) / 2.0) • http://hdf.ncsa.uiuc.edu/PMODELS/datatransform/ - 33 - HDF
- 30. Weather Research Forecast (WRF) Model • WRF – NCAR community standard model • HDF5 I/O module for NCAR’s WRF • HDF5-WRF parallel I/O studies – Improved performance for computations with large I/O • Sequential HDF5-WRF studies – Compression can save disk space • See the poster • And see http://hdf.ncsa.uiuc.edu/apps/WRF-ROMS - 34 - HDF
- 31. netCDF-HDF Project • Enhanced NetCDF-4 Interface to HDF5 – Combine features of netCDF and HDF5 – Take advantage of their separate strengths • Collaboration between NCSA and Unidata • See poster: “Merging the netCDF and HDF5 libraries to achieve gains in performance and interoperability” - 35 - HDF
- 32. OPeNDAP – netCDF – HDF5 • OPeNDAP – A system for the transmitting data across the Internet – Supports selection of data using constraint expressions – Can translate data from one format to another • NetCDF and HDF5 – Formats of major interest to the OPeNDAP community • All three are in heavy use in the earth sciences • So the question is … - 36 - HDF
- 33. ne ne 5 F5 DF HD H Are the planets finally aligned? tC tC F DF D N eN Pe OP O P P A DA D - 37 - To harmonize OPeNDAP netCDF HDF5 ? HDF
- 34. OpenDAP/netCDF/HDF5 Harmonization • Opportunity – – – – Unidata is creating netcdf-4 Existing OPeNDAP work with netcdf and HDF5 OPeNDAP project working on a new spec (4.0) John Caron working on new java-netCDF library (2.2) • Creates a "common data model" which is more-or-less a union of the 3 models. • But there are important differences – Different ecological niche – Some very different object types – So a union of all the models is unlikely - 38 - HDF
- 35. OpenDAP/netCDF/HDF5 Harmonization • Goal: map between the three models, and possibly tweak the models to better make them harmonize. • Tackle certain important differences – OPeNDAP Sequences • Hard to represent in the netCDF API • But seems like they might work in HDF5. – HDF5 attributes • Hard to represent in the DAP. • Also perhaps devise a formal mapping between the three models - 39 - HDF
- 36. Acknowledgements This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA grant NAG 5-2040 and NAG NCCS-599. Any opinions, findings, and conclusions or Thank you recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration. Other support provided by NCSA and other sponsors and agencies. (http://hdf.ncsa.uiuc.edu/acknowledge.html). HDF - 40 -
- 38. Information Sources • HDF website – http://hdf.ncsa.uiuc.edu/ • HDF5 Information Center – http://hdf.ncsa.uiuc.edu/HDF5/ • HDF Helpdesk – hdfhelp@ncsa.uiuc.edu • HDF users mailing list – hdfnews@ncsa.uiuc.edu - 42 - HDF
Editor's Notes
- #2: <number>
- #32: Indexing would: Enable apps conveniently to organize, find and view records of interest Enable apps have alternate views of data records
- #39: Could also extend and tweak the netCDF data model. Involves mapping between the HDF5 and netCDF models. OPeNDAP servers read netCDF and serve OPeNDAP OPeNDAP clients read OPeNDAP into a netCDF OPeNDAP clients read OPeNDAP into HDF5
- #40: Restrictions Will not actually merge the models, but at least clarify their relationships. Restrict ourselves to the data models, but have implementation and performance concerns lurking close at hand.