MATLAB Modernization on HDF5 1.10

1
© 2021 The MathWorks, Inc.
MATLAB Modernization on HDF5 1.10
Support for SWMR and VDS
and Cloud Data Access
Ellen Johnson
Senior Software Engineer, MathWorks
ESIP Summer 2021/HDF Workshop
July 23, 2021

2
Agenda
 Scientific data overview
 HDF5 interface
 What we’ve been doing
 What’s new in 21b
 Demo
 Performance and compatibility
 What’s in the future
 Wrap-up and Q&A
21b topics covered are available now to MATLAB users in
R2021b prerelease
Full R2021b release planned for September

3
Scientific Data in MATLAB
Scientific data formats
 HDF5, HDF4, HDF-EOS2
 NetCDF (with OPeNDAP)
 FITS, CDF, BIL, BIP, BSQ
Image file formats
 TIFF, JPEG, PNG, JPEG2000, HDR,
and more
Vector data file formats
 ESRI Shapefiles, KML, GPS
and more
Raster data file formats
 GeoTIFF, NITF, USGS and SDTS DEM,
NIMA DTED, and more
Web Map Service (WMS)

4
Working with HDF5 in MATLAB
MATLAB has two HDF5 interfaces
 High-level (HL) : Ease-of-use, less control
 Low-level (LL) : Wraps HDF5 C library, more control
Interface Function
High-level h5create, h5read,
h5write, h5disp, etc.
Low-level H5F.open
H5F.start_swmr_write
H5D.read
H5G.create
H5P.select_hyperslab
H5P.set_layout
H5P.set_virtual
H5S.create_simple
H5T.refresh
and over 300 more

5
What We’ve Been Up To
 Upgraded to HDF5 1.8.12
 Support for reading datasets with Dynamically Loaded Filters
 attempted upgrade to 1.10.2…performance regressions
 Support for HDF5 remote data access
 Wrote in-house Virtual File Driver
– S3 and Azure: Read/Write
– Hadoop: Read-only
– Enabled for all HL and LL functions
 Support for MAT-file v7.3 remote save/load
 more work on upgrading to 1.10.7…still regressions, but devised a solution

6
What’s New in R2021b
 MATLAB customer-facing HDF5 interface now uses HDF5 1.10.7
 New functions in low-level interface for:
– Single-Writer/Multiple-Reader
– Virtual Dataset
– Fine Tuning the Metadata Cache
– Partial Edge Chunk
 Modified existing functions for 1.10.7
 Shipping binaries for both 1.10.7 and 1.8.12 (Interim solution)
– 1.10.7 for MATLAB HDF5 interface
– 1.8.12 for MAT-file v.7.3 to avoid 1.10 regressions
– Consulting with THG and MathWorks teams on solution
 Goal: Ship one version and stay current with HDF5 releases

7
Functional Details in R2021b
 New functions added to LL interface
– Added ~30 new functions across the 16 APIs
– Provides fine-grained control of SWMR, VDS, Partial Edge Chunk, Metadata Cache
 Modified existing functions to work with 1.10.7
– H5F.open (for SWMR)
– H5P.set_layout (for VDS)
– H5R.dereference (for 1.10 signature)
– H5P.set_libver_bounds (for new high/low values)
 h5read, h5disp, h5info can access Virtual Datasets
whether stored locally or cloud
– S3, Azure, Hadoop

8
New Functions Mapped to HDF5 Features
HDF5 Feature MATLAB Function
SWMR H5F.start_swmr_write
H5O.disable_mdc_flushes
H5O.enable_mdc_flushes
H5O.are_mdc_flushes_disabled
VDS H5P.set_virtual H5P.get_virtual_dsetname H5P.set_virtual_view
H5P.get_virtual_count H5P.get_virtual_filename H5P.get_virtual_view
H5P.get_virtual_vspace H5P.set_virtual_printf_gap H5S.is_regular_hyperslab
H5P.get_virtual_srcspace H5P.gset_virtual_printf_gap H5S.get_regular_hyperslab
Fine Tuning the MDC H5F.get_metadata_read_retry_info H5D.flush H5O.flush
H5P.get_metadata_read_attempts H5D.refresh H5O.refresh
H5P.set_metadata_read_attempts H5G.flush H5T.flush
H5F.get_intent H5G.refresh H5T.refresh
Partial Edge Chunk H5P.get_chunk_opts
H5P.set_chunk_opts

9
Demo: Using SWMR with VDS in MATLAB with parpool
Mix of local and remote data access

10
Performance
Performance benchmarks with 1.10.7 vs 1.8.12
Improvements
– h5write, h5create, many low-level functions: minimal/moderate improvements
Regressions
– h5info: Substantial regressions with highly-nested groups with small datasets
– Working with THG to determine if same issue as MAT-file v7.3
Future performance work
– Optimize our HDF5 codebase (have identified target areas)
– Adding more workflow-based performance tests

11
Compatibility
Linux-only: Filter plugins with calls to core HDF5 library need to be rebuilt
with our shipping symbol-versioned HDF5 1.10.7 binary
– To avoid issues due to symbol collisions
– Option 1: Rebuild plugin with /matlab/bin/glnxa64/libhdf5.so.103.3.0
– Option 2: Build 1.10.7 using our GNU export map, then rebuild plugin with this binary.
– Will provide instructions in documentation and File Exchange
Interim solution until we ship one version again
H5P.set_libver_bounds
– low/high = latest/latest will create incompatible files with earlier MATLAB versions

12
Future Work Under Consideration
Highest priority
 Ship one HDF5 version (Linux plugin workaround no longer required)
 Support for writing datasets using Dynamically Loaded Filters
 Better experience for working with filter plugins
 Upgrade to HDF5 1.14 when available, evaluate new features to support
Watch-and-wait
 High-level support for creating VDS, controlling SWMR settings
 Support for other 1.10 features (Page Buffering, File Space Management)
 Looking for community feedback

13
Wrap-up and Q&A
 MATLAB now current with latest HDF5 version on 1.10 branch
 New SWMR and VDS capabilities
 Linux Filter Plugin compatibility
Please try out our new functions in R2021b prerelease
We love hearing feedback – it helps us improve our products!
Reach out to us with any questions or wish-lists!
- ellenj@mathworks.com

14
© 2021 The MathWorks, Inc.
Acknowledgements
Thank You!
 GEBCO Gridded Bathymetry Data: https://www.gebco.net/data_and_products/gridded_bathymetry_data/
GEBCO Compilation Group (2020) GEBCO 2020 Grid (doi:10.5285/a29c5465-b138-234d-e053-6c86abc040b9)
 The HDF Group: www.hdfgroup.com
 HDF5 VDS RFC: https://portal.hdfgroup.org/display/HDF5/RFC+HDF5+Virtual+Dataset

MATLAB Modernization on HDF5 1.10

More Related Content

What's hot (20)

Similar to MATLAB Modernization on HDF5 1.10 (20)

More from The HDF-EOS Tools and Information Center (17)

Recently uploaded (20)

MATLAB Modernization on HDF5 1.10

Editor's Notes