Introduction to HDF5
Download as PPT, PDF0 likes379 views
HDF5 is an open-source file format designed for high-volume or complex data and provides an organized data model, with hierarchical data structure. It allows flexible and efficient storage, I/O, and long-term data preservation, featuring a variety of data types, spaces, and metadata attributes associated with data objects. The document outlines the HDF5 technology platform, programming models, and basic API functions to interact with HDF5 files.
![Code: Create a Dataset
1
2
.
.
.
.
5
6
7
hid_t
hsize_t
herr_t
file_id, dataset_id, dataspace_id;
dims[2];
status;
file_id = H5Fcreate (”file.h5", a dataspace
Define H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT);
dims[0] = 4;
current dims
rank
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);
8
dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE,
dataspace_id, H5P_DEFAULT,
H5P_DEFAULT,
H5P_DEFAULT);
9 status = H5Dclose (dataset_id);
10 status = H5Sclose (dataspace_id);
11 status = H5Fclose (file_id);
September 28-30, 2010
HDF/HDF-EOS Workshop XIV
42
www.hdfgroup.org](https://image.slidesharecdn.com/introtohdf5ws14-140217084431-phpapp01/85/Introduction-to-HDF5-41-320.jpg)
![Code: Create a Dataset
1
.
.
.
.
.
.
.
8
hid_t
hsize_t
herr_t
file_id, dataset_id, dataspace_id;
dims[2];
status;
file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT);
dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);
Where to put it
Datatype
dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE,
dataspace_id, H5P_DEFAULT,H5P_DEFAULT,
H5P_DEFAULT);
Size & shape
September 28-30, 2010
Properties
(Link Creation, Dataset
Creation and Access)
HDF/HDF-EOS Workshop XIV
43
www.hdfgroup.org](https://image.slidesharecdn.com/introtohdf5ws14-140217084431-phpapp01/85/Introduction-to-HDF5-42-320.jpg)
![Code: Create a Dataset
1
2
3
hid_t
hsize_t
herr_t
file_id, dataset_id, dataspace_id;
dims[2];
status;
4
file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT);
5
6
7
dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);
8
dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE,
dataspace_id, H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
Terminate access to
9 status = H5Dclose (dataset_id);
10 status = H5Sclose (dataspace_id);
11 status = H5Fclose (file_id);
September 28-30, 2010
HDF/HDF-EOS Workshop XIV
dataspace, dataset, file
44
www.hdfgroup.org](https://image.slidesharecdn.com/introtohdf5ws14-140217084431-phpapp01/85/Introduction-to-HDF5-43-320.jpg)
Introduction to HDF5
- 1. The HDF Group Introduction to HDF5 Mike McGreevy The HDF Group The 14th HDF and HDF-EOS Workshop September 28-30, 2010 September 28-30, 2010 HDF/HDF-EOS Workshop XIV 1 www.hdfgroup.org
- 2. What is HDF5? • Open file format • Designed for high volume or complex data • Open source software • Works with data in the format • A data model • Structures for data organization and specification September 28-30, 2010 HDF/HDF-EOS Workshop XIV 2 www.hdfgroup.org
- 3. HDF = Hierarchical Data Format • HDF4 is the first HDF • Originally called HDF; last major release was version 4 • HDF5 benefits from lessons learned with HDF4 • Changes to file format, software, and data model • HDF5 and HDF4 are different • No plans for an HDF6! September 28-30, 2010 HDF/HDF-EOS Workshop XIV 3 www.hdfgroup.org
- 4. HDF5 is like … September 28-30, 2010 HDF/HDF-EOS Workshop XIV 4 www.hdfgroup.org
- 5. HDF5 is designed … • for high volume and/or complex data • for every size and type of system (portable) • for flexible, efficient storage and I/O • to enable applications to evolve in their use of HDF5 and to accommodate new models • to support long-term data preservation September 28-30, 2010 HDF/HDF-EOS Workshop XIV 5 www.hdfgroup.org
- 6. HDF5 Technology Platform • HDF5 data model • The “building blocks” for data organization and specification • HDF5 software • Library, language interfaces, tools • HDF5 file format • Bit-level organization of HDF5 file September 28-30, 2010 HDF/HDF-EOS Workshop XIV 6 www.hdfgroup.org
- 7. HDF5 Data Model Dataset Group Attribute Link HDF5 Objects Datatype Dataspace File a.k.a. HDF5 Abstract Data Model a.k.a. HDF5 Logical Data Model September 28-30, 2010 HDF/HDF-EOS Workshop XIV 7 www.hdfgroup.org
- 8. HDF5 File An HDF5 file is a container that holds data objects. September 28-30, 2010 HDF/HDF-EOS Workshop XIV lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 8 www.hdfgroup.org
- 9. HDF5 Dataset HDF5 Datatype Integer 32bit LE HDF5 Dataspace Rank Dimensions 3 Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements • HDF5 datasets organize and contain “raw data values”. • HDF5 datatypes describe individual data elements. • HDF5 dataspaces describe the logical layout of the data elements. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 9 www.hdfgroup.org
- 10. HDF5 Dataspaces • Describe the logical layout of the elements in an HDF5 dataset • NULL • no elements • Scalar • single element • Simple array (most common) • multiple elements organized in a rectangular array • rank = number of dimensions • dimension sizes = number of elements in each dimension • maximum number of elements in each dimension • may be fixed or unlimited September 28-30, 2010 HDF/HDF-EOS Workshop XIV 10 www.hdfgroup.org
- 11. HDF5 Dataspaces Two roles: Dataspace contains spatial information (logical layout) about a dataset stored in a file • Rank and dimensions • Permanent part of dataset definition Rank = 2 Dimensions = 4x6 Partial I/0: Dataspace describes application’s data buffer and data elements participating in I/O Rank = 1 Dimension = 10 September 28-30, 2010 HDF/HDF-EOS Workshop XIV 11 www.hdfgroup.org
- 12. HDF5 Dataset & Dataspace Dim_2 = 7 HDF5 Dataspace Rank Dimensions 3 Specifications for array dimensions Multi-dimensional array of identically typed data elements • HDF5 datasets organize and contain “raw data values”. • HDF5 dataspaces describe the logical layout of the data elements. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 12 www.hdfgroup.org
- 13. HDF5 Datatypes • Describe individual data elements in an HDF5 dataset • Wide range of datatypes supported • Integer • Float • • • • • Unsigned User-defined (e.g., 13-bit integer) Variable length types (e.g., strings) Compound (similar to C structs) Many more … September 28-30, 2010 HDF/HDF-EOS Workshop XIV 13 www.hdfgroup.org
- 14. HDF5 Dataset 3 5 12 Datatype: 32-bit Integer Dataspace: Rank = 2 Dimensions = 5 x 3 September 28-30, 2010 HDF/HDF-EOS Workshop XIV 14 www.hdfgroup.org
- 15. HDF5 Dataset with Compound Datatype 3 5 V int16 char int32 V V V V V V V V 2x3x2 array of float32 Compound Datatype: Dataspace: September 28-30, 2010 Rank = 2 Dimensions = 5 x 3 HDF/HDF-EOS Workshop XIV 15 www.hdfgroup.org
- 16. HDF5 Dataset & Datatype HDF5 Datatype Integer 32bit LE Specifications for single data element Multi-dimensional array of identically typed data elements • HDF5 datasets organize and contain “raw data values”. • HDF5 datatypes describe individual data elements. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 16 www.hdfgroup.org
- 17. HDF5 Dataset HDF5 Datatype Integer 32bit LE HDF5 Dataspace Rank Dimensions 3 Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements • HDF5 datasets organize and contain “raw data values”. • HDF5 datatypes describe individual data elements. • HDF5 dataspaces describe the logical layout of the data elements. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 17 www.hdfgroup.org
- 18. HDF5 Data Model: Are we there yet? HDF5 Objects Group and Link Attribute Dataspace Datatype Dataset File September 28-30, 2010 HDF/HDF-EOS Workshop XIV 18 www.hdfgroup.org
- 19. HDF5 Attributes • Typically contain user metadata • Have a name and a value • Are associated with HDF5 objects. • Value is described by a datatype and a dataspace • analogous to a dataset September 28-30, 2010 HDF/HDF-EOS Workshop XIV 19 www.hdfgroup.org
- 20. HDF5 Groups and Links HDF5 groups and links organize data objects. / Experiment Notes: Serial Number: 99378920 Date: 3/13/09 Configuration: Standard 3 Every HDF5 file has a root group SimOut Viz Parameters 10;100;1000 lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Timestep 36,000 September 28-30, 2010 HDF/HDF-EOS Workshop XIV 20 www.hdfgroup.org
- 21. HDF5 Technology Platform • HDF5 data model • The “building blocks” for data organization and specification • HDF5 software • Library, language interfaces, tools September 28-30, 2010 HDF/HDF-EOS Workshop XIV 21 www.hdfgroup.org
- 22. HDF5 Home Page HDF5 home page: http://hdfgroup.org/HDF5/ • Latest release: HDF5 1.8.5 (1.8.6 coming in October!) HDF5 source code: • • Written in C, and includes optional C++, Fortran 90 APIs, and High Level APIs Contains command-line utilities (h5dump, h5repack, h5diff, ..) and compile scripts HDF5 pre-built binaries: • When possible, include C, C++, F90, and High Level libraries. Check ./lib/libhdf5.settings file. • Built with and require the SZIP and ZLIB external libraries September 28-30, 2010 HDF/HDF-EOS Workshop XIV 22 www.hdfgroup.org
- 23. HDF5 API and Applications Applications EOS Application Domain Data Objects EOS library MATLAB … HDF5 Library Storage September 28-30, 2010 HDF/HDF-EOS Workshop XIV 23 www.hdfgroup.org
- 24. HDF5 Library Tools HDF5 Software Layers & Storage API … Language Interfaces C, Fortran, C++ Internals Virtual File Layer High Level APIs h5dump tool h5repack tool HDFview tool Java Interface HDF5 Data Model Objects Tunable Properties Groups, Datasets, Attributes, … Chunk Size, I/O Driver, … Memory Mgmt Datatype Conversion Filters Split Files Posix I/O Chunked Storage Version and so on… Compatibility Custom MPI I/O Storage I/O Drivers HDF5 File Format September 28-30, 2010 File Split Files HDF/HDF-EOS Workshop XIV File on Parallel Filesystem 24 Other www.hdfgroup.org
- 25. Useful Tools For New Users h5dump: Tool to “dump” or display contents of HDF5 files h5cc, h5c++, h5fc: Scripts to compile applications HDFView: Java browser to view HDF4 and HDF5 files http://www.hdfgroup.org/hdf-java-html/hdfview/ September 28-30, 2010 HDF/HDF-EOS Workshop XIV 25 www.hdfgroup.org
- 26. Introduction to HDF5 Programming Model and APIs September 28-30, 2010 HDF/HDF-EOS Workshop XIV 26 www.hdfgroup.org
- 27. General Programming Paradigm • Object is opened or created • Object is accessed, possibly many times • Object is closed • Properties of object are optionally defined Creation properties Access properties September 28-30, 2010 HDF/HDF-EOS Workshop XIV 27 www.hdfgroup.org
- 28. Order of Operations • An order is imposed on operations by argument dependencies For Example: A file must be opened before a dataset -becausethe dataset open call requires a file handle as an argument. • Objects can be closed in any order. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 28 www.hdfgroup.org
- 29. The General HDF5 API • Currently C, Fortran 90, Java, and C++ bindings. • C routines begin with prefix H5? ? is a character corresponding to the type of object the function acts on Example Functions: H5D : Dataset interface e.g., H5Dread H5F : File interface e.g., H5Fopen H5S : dataSpace interface e.g., H5Sclose September 28-30, 2010 HDF/HDF-EOS Workshop XIV 29 www.hdfgroup.org
- 30. HDF5 Defined Types For portability, the HDF5 library has its own defined types: hid_t: object identifiers (native integer) hsize_t: size used for dimensions (unsigned long or unsigned long long) herr_t: function return value hvl_t: variable length datatype Note: This is not an exhaustive list! For C, include hdf5.h in your HDF5 application. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 30 www.hdfgroup.org
- 31. The HDF5 API • For flexibility, the API is extensive Victronix Swiss Army Cybertool 34 300+ functions • This can be daunting… but there is hope A few functions can do a lot Start simple Build up knowledge as more features are needed September 28-30, 2010 HDF/HDF-EOS Workshop XIV 31 www.hdfgroup.org
- 32. Basic Functions H5Fcreate (H5Fopen) create (open) File H5Screate_simple/H5Screate H5Dcreate (H5Dopen) H5Dread, H5Dwrite H5Dclose create (open) Dataset access Dataset close Dataset H5Sclose close dataSpace H5Fclose September 28-30, 2010 create dataSpace close File HDF/HDF-EOS Workshop XIV 32 www.hdfgroup.org
- 33. Other Common Functions DataSpaces: H5Sselect_hyperslab (Partial I/O) H5Sselect_elements (Partial I/O) H5Dget_space Groups: H5Gcreate, H5Gopen, H5Gclose Attributes: H5Acreate, H5Aopen_name, H5Aclose, H5Aread, H5Awrite Property lists: September 28-30, 2010 H5Pcreate, H5Pclose H5Pset_chunk, H5Pset_deflate HDF/HDF-EOS Workshop XIV 33 www.hdfgroup.org
- 34. The HDF Group Example HDF5 Code September 28-30, 2010 HDF/HDF-EOS Workshop XIV 35 www.hdfgroup.org
- 35. Steps to Create a File 1. Decide on properties the file should have and create them if necessary: • Creation properties • Access properties • We will use Default properties. 2. Create the file 3. Close the file and the property lists, as needed September 28-30, 2010 HDF/HDF-EOS Workshop XIV 36 www.hdfgroup.org
- 36. Code: Create a File hid_t herr_t file_id; status; file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); status = H5Fclose (file_id); “/” (root) Note: Return codes not checked for errors in code samples. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 37 www.hdfgroup.org
- 37. Steps to Create a Dataset 1. Define dataset characteristics a) Datatype – integer b) Dataspace - 4x6 c) Properties if needed, or use H5P_DEFAULT 2. Decide where to put it “/” (root) 2. Group or root group 3. Create dataset in file 4. Close everything September 28-30, 2010 HDF/HDF-EOS Workshop XIV A 38 www.hdfgroup.org
- 38. HDF5 Pre-defined Datatype Identifiers HDF5 defines* set of Datatype Identifiers per HDF5 session. For example: C Type HDF5 File Type HDF5 Memory Type int H5T_STD_I32BE H5T_STD_I32LE H5T_NATIVE_INT float H5T_IEEE_F32BE H5T_IEEE_F32LE H5T_NATIVE_FLOAT double H5T_IEEE_F64BE H5T_IEEE_F64LE H5T_NATIVE_DOUBLE * Value of datatype is NOT fixed September 28-30, 2010 HDF/HDF-EOS Workshop XIV 39 www.hdfgroup.org
- 39. Pre-defined File Datatype Identifiers Examples: H5T_IEEE_F64LE H5T_STD_I32LE Architecture* Eight-byte, little-endian, IEEE floating-point Four-byte, little-endian, signed two's complement integer Programming Type NOTE: What you see in the file. Name is the same everywhere and explicitly defines a datatype. *STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…” September 28-30, 2010 HDF/HDF-EOS Workshop XIV 40 www.hdfgroup.org
- 40. Pre-defined Native Datatypes Examples of predefined native types in C: H5T_NATIVE_INT H5T_NATIVE_FLOAT H5T_NATIVE_UINT H5T_NATIVE_LONG H5T_NATIVE_CHAR (int) (float ) (unsigned int) (long ) (char ) NOTE: Memory types. Different for each machine. Used for reading/writing. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 41 www.hdfgroup.org
- 41. Code: Create a Dataset 1 2 . . . . 5 6 7 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; file_id = H5Fcreate (”file.h5", a dataspace Define H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); dims[0] = 4; current dims rank dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 8 dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); 9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id); September 28-30, 2010 HDF/HDF-EOS Workshop XIV 42 www.hdfgroup.org
- 42. Code: Create a Dataset 1 . . . . . . . 8 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); Where to put it Datatype dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT,H5P_DEFAULT, H5P_DEFAULT); Size & shape September 28-30, 2010 Properties (Link Creation, Dataset Creation and Access) HDF/HDF-EOS Workshop XIV 43 www.hdfgroup.org
- 43. Code: Create a Dataset 1 2 3 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; 4 file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); 5 6 7 dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 8 dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); Terminate access to 9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id); September 28-30, 2010 HDF/HDF-EOS Workshop XIV dataspace, dataset, file 44 www.hdfgroup.org
- 44. Example Code - H5Dwrite Dataset ID from H5Dcreate/H5Dopen Memory Datatype status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL,H5S_ALL, H5P_DEFAULT, wdata); September 28-30, 2010 HDF/HDF-EOS Workshop XIV 45 www.hdfgroup.org
- 45. Partial I/O status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,wdata); Memory Dataspace H5S_ALL H5S_ALL File Dataspace (disk) To Modify Dataspace: H5Sselect_hyperslab H5Sselect_elements September 28-30, 2010 HDF/HDF-EOS Workshop XIV 46 www.hdfgroup.org
- 46. Example Code – H5Dwrite status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata); Data Transfer Property List (MPI I/O, Transformations,…) September 28-30, 2010 HDF/HDF-EOS Workshop XIV 47 www.hdfgroup.org
- 47. Example Code – H5Dread status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata); September 28-30, 2010 HDF/HDF-EOS Workshop XIV 48 www.hdfgroup.org
- 48. Steps to Create a Group 1. Decide where to put it – “root group” 2. Define properties or use H5P_DEFAULT 3. Create group in file. 4. Close the group. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 51 www.hdfgroup.org
- 49. Example: Create a Group “/” (root) A B 4x6 array of integers file.h5 September 28-30, 2010 HDF/HDF-EOS Workshop XIV 52 www.hdfgroup.org
- 50. Code: Create a Group hid_t file_id, group_id; ... /* Open “file.h5” */ file_id = H5Fopen (“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT); /* Create group "/B" in file. */ group_id = H5Gcreate (file_id,"B", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); /* Close group and file. */ status = H5Gclose (group_id); status = H5Fclose (file_id); September 28-30, 2010 HDF/HDF-EOS Workshop XIV 53 www.hdfgroup.org
- 51. HDF5 Tutorial and Examples HDF5 Tutorial: http://www.hdfgroup.org/HDF5/Tutor/ HDF5 Example Code: http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/ September 28-30, 2010 HDF/HDF-EOS Workshop XIV 54 www.hdfgroup.org
- 52. HDF5 Technology Platform • HDF5 data model • The “building blocks” for data organization and specification • HDF5 software • Library, language interfaces, tools • HDF5 file format • Bit-level organization of HDF5 file September 28-30, 2010 HDF/HDF-EOS Workshop XIV 55 www.hdfgroup.org
- 53. HDF5 File Format • Defined by the HDF5 File Format Specification. http://www.hdfgroup.org/HDF5/doc/H5.format.html • Specifies the bit-level organization of an HDF5 file on storage media. • HDF5 library adheres to the File Format, so for the most part basic users do not need to know the guts of this information. September 28-30, 2010 HDF/HDF-EOS Workshop XIV 56 www.hdfgroup.org
- 54. HDF5 Technology Platform • HDF5 data model • The “building blocks” for data organization and specification • HDF5 software • Library, language interfaces, tools • HDF5 file format • Bit-level organization of HDF5 file September 28-30, 2010 HDF/HDF-EOS Workshop XIV 57 www.hdfgroup.org
- 55. The HDF Group Thank You! September 28-30, 2010 HDF/HDF-EOS Workshop XIV 58 www.hdfgroup.org
- 56. The HDF Group Questions/comments? September 28-30, 2010 HDF/HDF-EOS Workshop XIV 59 www.hdfgroup.org