HDF for the Cloud
Download as PPTX, PDF1 like744 views
This document discusses strategies for storing and accessing HDF5 data files in cloud object storage like Amazon S3. It describes an HDF5 Virtual File Driver (VFD) developed by The HDF Group that allows reading HDF5 files directly from S3 without downloading. For better performance, the document recommends optimizing HDF5 files stored in S3 by using chunking, compression, and aggregating smaller files. It also introduces the HDF Cloud Schema which maps HDF5 objects to individual object storage objects for parallel access.
![8Dataset JSON Example
creationProperties contains HDF5 dataset
creation property list settings.
Id is the objects UUID.
Layout represents HDF5 dataspace.
Root points back to the root group
Created & lastModified are timestamps
type represents HDF5 datatype.
attributes holds a list of HDF5 attribute
JSON objects.
{
"creationProperties": {},
"id": "d-9a097486-58dd-11e8-a964-
0242ac110009",
"layout": {"dims": [10], "class":
"H5D_CHUNKED"},
"root": "g-952b0bfa-58dd-11e8-a964-
0242ac110009",
"created": 1526456944,
"lastModified": 1526456944,
"shape": {"dims": [10], "class":
"H5S_SIMPLE"},
"type": {"base": "H5T_STD_I32LE",
"class": "H5T_INTEGER"},
"attributes": {}
}](https://image.slidesharecdn.com/jxr2-180802015331/85/HDF-for-the-Cloud-8-320.jpg)
HDF for the Cloud
- 1. HDF for the Cloud 1 John Readey
- 2. The HDF5 data format 2 âą Established 20 years ago the HDF5 file format is the most commonly used format in Earth Science âą Note: NetCDF4 files are actually HDF5 âunder the hoodâ âą HDF5 was designed with the (somewhat contradictory) goals of: âą Archival format â data that can stored for decades âą Analysis Ready -- data that can be directly utilized for analytics (no conversion needed) âą Thereâs a rich set of tools and language SDKs: âą C/C++/Fortran âą Python âą Java, etc.
- 3. HDF5 File Format meets the Cloud 3 âą Storing large HDF5 collection on AWS is almost always about utilizing S3: âą Cost effective âą Redundant âą Sharable âą Itâs easy enough to store HDF5 files as S3 objects, but these files canât be read using the HDF5 library (which is expecting a POSIX filesystem) âą Experience using FUSE to read from S3 using HDF5Library has not tended to work so well âą In practice users have been left with copying files to local disk first âą This has led to interest in alternative formats such as Zarr, TileDB, and our own HSDS S3 Storage Schema (more on that later)
- 4. HDF5 meets S3 halfway⊠4 âą For many years the HDF5 library has supported VFDs âVirtual File Driverâ âą VFDs are low-level plugins that can replace the standard POSIX IO methods with anything the developer of the VFD would like âą The HDF Group has developed a VFD specifically for S3 that will be included in the next library release (coming soon!) âą How it works: Each POSIX read call is replaced with a S3 Range GET âą Features: âą Can read any HDF5 file (write is not supported) âą No changes to the public API âą Compatible with higher-level libraries (h5py, netcdf, xarray, etc.) âą This is a first release and there are some ideas for improving performance in subsequent releases âą It will be very helpful to come up with an objective set of benchmarks to compare performance between S3VFD, HSDS, Zarr, etc.
- 5. Cloud Optimized HDF âą For anyone putting HDF5 files on S3 for in-place reading, there are a few things that can be done to improve performance when accessed using the S3VFD (or FUSE) âą Most of these optimizations can be done using existing tools (e.g. h5repack) âą A Cloud Optimized HDF5 files is still an HDF5 file and can be downloaded and read with native VFD if desired âą Initial Proposal (likely to be revised based on testing): âą Use chunking for datasets larger than 1MB âą Use âbrick styleâ chunk layouts (enable slicing via any dimension) âą Use readily available compression filters âą Pack metadata in front of file âą Aggregate smaller files into larger ones
- 6. HDF Server 6 âą HSDS (now HDF Kita Server) is a REST based service for HDF data developed by the HDF Group âą Think of it as HDF gone cloud native. ï âą HSDS Features: âą Runs as a set of containers on Kubernetes â so can scale beyond one machine âą Requests can be parallelized across multiple containers âą Feature compatible with the HDF library but is independent code base âą Supports multiple readers/writers âą Uses S3 as data store âą Available now as part of HDF Kita Lab (our hosted Jupyter environment): https://hdflab.hdfgroup.org âą Will be available on AWS Marketplace soon
- 7. HDF Cloud Schema Big Idea: Map individual HDF5 objects (datasets, groups, chunks) as Object Storage Objects âą Limit maximum storage object size âą Support parallelism for read/write âą Only data that is modified needs to be updated âą Multiple clients can be reading/updating the same âfileâ Legend: âą Dataset is partitioned into chunks âą Each chunk stored as an S3 object âą Dataset meta data (type, shape, attributes, etc.) stored in a separate object (as JSON text) How to store HDF5 content in S3? Each chunk (heavy outlines) get persisted as a separate object
- 8. 8Dataset JSON Example creationProperties contains HDF5 dataset creation property list settings. Id is the objects UUID. Layout represents HDF5 dataspace. Root points back to the root group Created & lastModified are timestamps type represents HDF5 datatype. attributes holds a list of HDF5 attribute JSON objects. { "creationProperties": {}, "id": "d-9a097486-58dd-11e8-a964- 0242ac110009", "layout": {"dims": [10], "class": "H5D_CHUNKED"}, "root": "g-952b0bfa-58dd-11e8-a964- 0242ac110009", "created": 1526456944, "lastModified": 1526456944, "shape": {"dims": [10], "class": "H5S_SIMPLE"}, "type": {"base": "H5T_STD_I32LE", "class": "H5T_INTEGER"}, "attributes": {} }
- 9. Schema Details 9 âą Key dispersal âą Objects are stored âflatâ â no hierarchy âą UUIDs have a 5 char hash added to the front âą Idea is the evenly distribute objects across S3 storage nodes to improve performance âą S3 partitions objects by first few characters of the key name âą Each storage node is limited to about 300 req/s âą Thereâs no list of chunks âą Chunk key is determined based on chunk position in the data space âą E.g. c-<uuid>_0_0_0 Is the corner chunk of a 3-dimensional dataset âą Chunk objects get created as needed on first write âą Schema is currently used just by HDF Server, but could just as easily be used directly by clients (assuming that writes donât conflict)
- 10. Supporting traditional HDF5 files 1 0 âą Downside of the HDF S3 Schema is that data needs be transmogrified âą Since the bulk of the data is usually the chunk data it makes sense to combine the ideas of the S3 Schema and S3VFD: âą Convert just the metadata of the source HDF5 file to the S3 Schema âą Store the source file as a S3 object âą For data reads, metadata provides offset and length into the HDF5 file âą S3 Range GET returns needed data âą This approach can be used either directly or with HDF Server âą Compared with the pure S3VFD approach, you reduce the number of S3 requests needed âą Work on supporting this is planned for later this year
- 11. References 1 1 âą HDF Schema: https://s3.amazonaws.com/hdfgroup/docs/obj_store_schema.pdf âą SciPy2017 talk: https://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf âą AWS Big Data Blog article: https://aws.amazon.com/blogs/big-data/power- from-wind-open-data-on-aws/ âą AWS S3 Performance guidelines: https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf- considerations.html
Editor's Notes
- #2: Many users of HDF5 are now migrating data archives to public or private cloud systems. The access approaches and performance characteristics of cloud storage are fundamentally different than traditional data storage systems because 1) the data are accessed over http and 2) the data are stored in an object store and identified using unique keys. There are many different ways to organize and access data in the cloud. The HDF Group is currently exploring and developing approaches that will facilitate migration to the cloud and support many existing HDF5 data access use cases. Our goal is to protect data providers and users from disruption as their data and applications are migrated to the cloud.
- #8: This idea has been kicking around for a while, but storing potentially millions of files on a Linux filesystem would be problematic. Using S3 as the storage vehicle is a natural fit since thereâs no limit to the number of objects in a bucket. With NREL weâve validated this approach to 50 TBâs of data over 27MM objects (see aws-big-data blog article: https://aws.amazon.com/blogs/big-data/power-from-wind-open-data-on-aws/ )