HDF5 Roadmap 2019-2020
Download as PPTX, PDF1 like857 views
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
HDF5 Roadmap 2019-2020
- 1. Conf-DDDD-IN HDF5 Roadmap 2019-2020 Summer ESIP 2019 This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C. This document does not contain technology or Technical Data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. Elena Pourmal The HDF Group EED2 Team Lead epourmal@hdfgroup.org
- 2. Conf-DDDD-IN 2 • New features in Hierarchical Data Format 5 (HDF5) Release 1.12.0 • Support for HDF5 versions 1.8 and 1.10 • Summary of HDF5 Roadmap Outline
- 3. Conf-DDDD-IN 3 HDF5 References V1 | V2 | temp ---- |----- |----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 HDF5 dataset may store references to other objects or references to selected elements of a dataset. References to objects or dataset regions stored in other files are coming in HDF5 1.12.0. A.h5 B.h5 Time step 36,000 Dataset in B.h5 stores references to datasets stored in A.h5
- 4. Conf-DDDD-IN 4 HDF5 Reference to Attribute V1 | V2 | temp ---- |----- |----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 HDF5 1.12.0 will allow to reference an attribute of a dataset or a group. Dataset in A.h5 stores references to the attributes in A.h5 and B.h5. A.h5 B.h5 Time step 36,000 Time step 100,000
- 5. Conf-DDDD-IN 5 • HDF5 1.12.0 enables HDF5 to store data on new storage using Virtual Object Layer (VOL) connectors and Virtual File Drivers (VFDs). • VOL connectors developed by The HDF Group and its collaborators are available from public repository https://bitbucket.hdfgroup.org/projects/HDF5VOL HDF5 New Architecture
- 6. Conf-DDDD-IN 6 HDF5 New Architecture (cont’d) S3 DAOS Object Store DAOS REST S3 AIO 1 1 – HDF5 Application Programming Interface
- 7. Conf-DDDD-IN 7 • HDF5 1.12.0 will have a VFD to access HDF5 file via Amazon Simple Storage Service (Amazon S3) • Requires minimum changes to the application code • h5dump and h5ls tools have a flag to specify the driver to access HDF5 file on S3 h5ls --vfd=ros3 https://s3.us-east-2.amazonaws.com/file.h5 S3 VFD
- 8. Conf-DDDD-IN 8 • Uses “range get” commands to get “bytes” from HDF5 file stored on S3 • New API to set up S3 VFD herr_t H5Pset_fapl_ros3(hid_t fapl_id, H5FD_ros3_fapl_t *fa) • Credentials are passed via parameter to the function • Demo S3 VFD (cont’d)
- 9. Conf-DDDD-IN 9 • HDF5 1.12.0 will have a VFD to access HDF5 file on Hadoop Distributed File System (HDFS). • New API to access HDF5 file on HDFS herr_t H5Pset_fapl_hdfs(hid_t fapl_id); • HDF5 command line tools with enabled HDFS VFD allows to extract metadata and raw data from HDF5 and netCDF4 files on HDFS, and use Hadoop streaming to collect data from multiple HDF5 files. • Demo HDFS VFD
- 10. Conf-DDDD-IN 10 • UTF-8 will be default string encoding instead of ASCII starting with HDF5 1.12.0 – Names of groups, datasets, attributes – Names of compound datatypes fields, enums, names of the files that are stored in HDF5 according to the File Format spec (VDS, VFDs family and split) have to be ASCII. New default for string encoding
- 11. Conf-DDDD-IN 11 • The HDF Group will continue support for HDF5 1.8.* and 1.10.* releases – Security patches – Critical bug fixes – Performance improvements – Thorough testing of HDF5 1.8.* for forward compatibility with HDF5 1.10.* and 1.12.0 • Files created by applications that are built with the HDF5 version later than 1.8.* and that do not use any new features of that version are readable by 1.8.* Support for HDF5 1.8 and 1.10
- 12. Conf-DDDD-IN 12 Today Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 2019 2020 HDF5 Roadmap 2021 2022 Q1 Fall 2019 1.8.22 HDF5 1.8 • Security patches • Forward Compatibility with 1.12 HDF5 1.10 HDF5 2.0 instead of 1.12.0? Storage beyond the current File Systems and HDF5 file format Nov 1.10.6 May 1.10.7 Nov 1.10.8 • Address degradation of performance between major HDF5 releases 1.6 – 1.10 • VDS HDF5 performance • Public performance benchmark test suit • Dynamically loaded VFDs • S3, HDFS, Spark VFD connectors July 2019 1.12.0 • VOL architecture • References • Parallel compression (enhanced) Nov 2019 1.12.1 • VOL plugins (DAOS) • VFD plugins • Dynamically loaded VFDs • S3, HDFS, SPARK VFD connectors May 1.10.9 • Address degradation of performance between major HDF5 releases 1.6 – 1.10 • Security updates • Bug fixes 2020-2021 1.12.X • Query and Indexing • Full SWMR • Mirror VFD • Provenance (Onion) VFD Summer 2020 1.8.23 • Security patches • Forward Compatibility with 1.12 HDF5 1.12
- 13. Conf-DDDD-IN 13 This work was supported by NASA/GSFC under Raytheon Co. contract number NNG15HZ39C. in partnership with