Advanced HDF5 Features

HDF5 Advanced Topics

Elena Pourmal
The HDF Group
The 15th HDF and HDF-EOS Workshop
April 17, 2012
April 17-19

HDF/HDF-EOS Workshop XV

1

Goal
• To learn about HDF5 features important for
writing portable and efficient applications using
H5Py

April 17-19


2

Outline
• Groups and Links
• Types of groups and links
• Discovering objects in an HDF5 file

• Datasets
• Datatypes
• Partial I/O
• Other features
• Extensibility
• Compression

April 17-19


3

GROUPS AND LINKS

April 17-19


4

Groups and Links
• Groups are containers for links (graph edges)
• Links were added in 1.8.0
• Warning: Many APIs in H5G interface are
obsolete - use H5L interfaces to discover and
manipulate file structure

April 17-19


5

Groups and Links
HDF5 groups
and links
organize
data objects.

/

Experiment Notes:
Serial Number: 99378920
Date: 3/13/09
Configuration: Standard 3

Every HDF5 file
has a root group

SimOut

Viz
lat | lon | temp
----|-----|----12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6

Timestep
36,000

April 17-19, 2012


6

Parameters
10;100;1000

Example h5_links.py
Different kinds of
links

/

links.h5

A

B
dangling
a

soft

a

External

Dataset can be “reached”
using three paths
/A/a
/a
/soft
April 17-19, 2012


dset.h5

Dataset is in a different file
7

Example h5_links.py
Different kinds of
links

/

links.h5

A

B

dangling
a

soft

Hard links “A” and “B” were created when groups were created
Hard link “a” was added to the root group and points to an existing dataset
Soft link “soft” points to the existing dataset (cmp. UNIX alias)
Soft link “dangling” doesn’t point to any object
April 17-19, 2012


8

Links
• Name
• Example: “A”, “B”, “a”, “dangling”, “soft”
• Unique within a group; “/” are not allowed in names

• Type
• Hard Link
• Value is object’s address in a file
• Created automatically when object is created
• Can be added to point to existing object

• Soft Link
• Value is a string , for example, “/A/a”, but can be
anything
• Use to create aliases
April 17-19


9

Links (cont.)
• Type
• External Link
• Value is a pair of strings , for example, (“dset.h5”,
“dset” )
• Use to access data in other HDF5 files
• Example: For NPP data products geo-location information
may be in a separate file

April 17-19


10

Links Properties
• Links Properties
• ASCII or UTF-8 encoding for names
• Create intermediate groups
• Saves programming effort

• C example
lcpl_id = H5Pcreate(H5P_LINK_CREATE);
H5Gcreate (fid, "A/B", lcpl_id, H5P_DEFAULT, H5P_DEFAULT);

• Group “A” will be created if it doesn’t exist

April 17-19


11

Operations on Links
•
•
•
•
•
•

See H5L interface in Reference Manual
Create
Delete
Copy
Iterate
Check if exists

April 17-19


12

Operations on Links
• APIs available for C and Fortran
• Use dictionary operations in Python
• Objects associated with links ARE NOT affected
• Deleting a link removes a path to the object
• Copying a link doesn’t copy an object

April 17-19


13

Example h5_links.py
Link a in A is removed

/

links.h5

A

B
dangling
a

soft
External

Dataset can be “reached”
using one paths
/a

dset.h5

April 17-19, 2012


14

Example h5_links.py
Link a in root is
removed

/

links.h5

A

B
dangling
soft
External
dset.h5

Dataset is unreachable
April 17-19, 2012


15

Groups Properties
• Creation properties
• Type of links storage
• Compact (in 1.8.* versions)
• Used with a few members (default under 8)

• Dense (default behavior)
• Used with many (>16) members (default)

• Tunable size for a local heap
• Save space by providing estimate for size of the storage
required for links names

• Can be compressed (in 1.8.5 and later)
• Many links with similar names (XXX-abc, XXX-d, XXXefgh, etc.)
• Requires more time to compress/uncompress data
April 17-19


16

Groups Properties
• Creation properties
• Links may have creation order tracked and indexed
• Indexing by name (default)
• A, B, a, dangling, soft

• Indexing by creation order (has to be enabled)
• A, B, a, soft, dangling

• http://www.hdfgroup.org/ftp/HDF5/examples/exam
ples-by-api/api18-c.html

April 17-19


17

Discovering HDF5 file’s structure
• HDF5 provides C and Fortran 2003 APIs for
recursive and non-recursive iterations over the
groups and attributes
• H5Ovisit and H5Literate (H5Giterate)
• H5Aiterate

• Life is much easier with H5Py (h5_visita.py)
import h5py
def print_info(name, obj):
print name
for name, value in obj.attrs.iteritems():
print name+":", value
f = h5py.File('GATMO-SATMS-npp.h5', 'r+')
f.visititems(print_info)
f.close()
April 17-19


18

Checking a path in HDF5
• HDF5 1.8.8 provides HL C and Fortran 2003 APIs
for checking if paths exists
• H5LTvalid_path (h5ltvalid_path_f)
• Example: Is there an object with a path /A/B/C/d ?
• TRUE if there is a path, FALSE otherwise

April 17-19


19

Hints
• Use latest file format (see
H5Pset_libver_bound function in RM)
• Save space when creating a lot of groups in
a file
• Save time when accessing many objects
(>1000)
• Caution: Tools built with the HDF5 versions prirt
to 1.8.0 will not work on the files created with this
property

April 17-19


20

DATASETS

April 17-19


21

HDF5 Datatypes

April 17-19


22

HDF5 Datatypes
• Integer and floating point
• String
• Compound
• Similar to C structures or Fortran Derived Types

•
•
•
•
•

Array
References
Variable-length
Enum
Opaque

April 17-19


23

HDF5 Datatypes
• Datatype descriptions
• Are stored in the HDF5 file with the data
• Include encoding (e.g., byte order, size, and
floating point representation) and other
information to assure portability across

platforms
• See C, Fortran, MATLAB and Java
examples under
http://www.hdfgroup.org/ftp/HDF5/examples/

April 17-19


24

Data Portability in HDF5
Array of integers on Intel platform Array of long integers on SPARC64 platform
long is big-endian, 8 bytes
int is little-endian, 4 bytes

int

long

H5Dwrite

H5Dread

H5T_STD_I32LE

April 17-19


25

Data Portability in HDF5 (cont.)
We use native integer type to describe data in a
file
dset =
H5Dcreate(file,NAME,H5T_NATIVE_INT,…
Description of data in a buffer
H5Dwrite(dset,H5T_NATIVE_INT,…,buf);

H5Dread(dset,H5T_NATIVE_LONG,…, buf);
Description of data in a buffer; library will perform
Conversion from 4 byte LE to 8 byte BE integer

April 17-19


26

Hints
• Avoid datatype conversion if possible
• Store necessary precision to save space in
a file
• Starting with HDF5 1.8.7, Fortran APIs
support different kinds of integers and floats
(if Fortran 2003 feature is enabled)

April 17-19


27

HDF5 Strings

April 17-19


28

HDF5 Strings
• Fixed length
• Data elements has to have the same size
• Short strings will use more byte than needed
• Application responsible for providing buffers of the
correct size on read

• Variable length
• Data elements may not have the same size
• Writing/reading strings is “easy”; library handles
memory allocations

April 17-19


29

HDF5 Strings – Fixed-length
• Example h5_string.py(c,f90)

fixed_string = np.dtype('a10')
dataset = file.create_dataset("DSfixed",(4,), dtype=fixed_string)
data = ("Parting", ".is such", ".sweet", ".sorrow...")
dataset[...] = data

• Stores fours strings “Parting", ” .is such", ” .sweet",
”.sorrow…” in a dataset.
• Strings have length 10
• Python uses NULL padded strings (default)

April 17-19


30

HDF5 Strings
• Example h5_vlstring.py(c,f90)
str_type = h5py.new_vlen(str)
dataset = file.create_dataset("DSvariable",(4,), dtype=str_type)
data = ("Parting", " is such", " sweet", " sorrow...")
dataset[...] = data

• Stores fours strings “Parting", ” is such", ” sweet",
”sorrow…” in a dataset.
• Strings have length 7, 8, 6, 10

April 17-19


31

Hints
• Fixed length strings
• Can be compressed
• Use when need to store a lot of strings

• Variable-length strings
• Compression cannot be applied to data
• Use for attributes and a few strings if space is a
concern

April 17-19


32

HDF5 Compound Datatypes

April 17-19


33

HDF5 Compound Datatypes
• Compound types
• Comparable to C structures or Fortran 90
Derived Types
• Members can be of any datatype
• Data elements can written/read by a single field
or a set of fields

April 17-19


34

Creating and Writing Compound Dataset
• Example h5_compound.py(c,f90)
• Stores four records in the dataset
Orbit
integer

Location
string

Temperature (F)
64-bit float

Pressure (inHg)
64-bit-float

1153

Sun

53.23

24.57

1184

Moon

55.12

22.95

1027

Venus

103.55

31.33

1313

Mars

1252.89

84.11

April 17-19


35

Creating and Writing Compound Dataset
comp_type = np.dtype([('Orbit’,'i'),('Location’,np.str_, 6),
….)
dataset = file.create_dataset("DSC",(4,), comp_type)
dataset[...] = data

Note for C and Fortran2003 users:
• You’ll need to construct memory and file datatypes
• Use HOFFSET macro instead of calculating offset by hand.
• Order of H5Tinsert calls is not important if HOFFSET is used.

April 17-19


36

Reading Compound Dataset
f = h5py.File('compound.h5', 'r')
dataset = f ["DSC"]
….
orbit = dataset['Orbit']
print "Orbit: ", orbit
data = dataset[...]
print data
….
print dataset[2, 'Location']

April 17-19


37

Fortran 2003
• HDF5 Fortran library 1.8.8 with Fortran 2003
enabled has the same capabilities for writing
derived types as C library
• H5OFFSET function
• No need to write/read by fields as before

April 17-19


38

Hints
• When to use compound datatypes?
• Application needs access to the whole record

• When not to use compound datatypes?
• Application needs access to specific fields often
• Store the field in a dataset

/

/
DSC

Pressure

Orbit

Location
Temperature
April 17-19


39

HDF5 Reference Datatypes

April 17-19


40

References to Objects and Dataset Regions

/

Test Data

Viz
References to HDF5
Objects

References to dataset regions

.

Group
Image 2…..
Image 3…..

April 17-19, 2012


41

.

Reference Datatypes
• Object Reference
• Unique identifier of an object in a file
• HDF5 predefined datatype
H5T_STD_REG_OBJ
• Dataset Region Reference
• Unique identifier to a dataset + dataspace
selection
• HDF5 predefined datatype
H5T_STD_REF_DSETREG

April 17-19


42

Conceptual view of HDF5 NPP file
XML User’s Block

Product Group

Root - /

Agg
Reference
Object

Data
Gran n

Reference
Region

Reference
Region

43

NPP HDF5 file in HDFView

April 17-19


44

HDF5 Object References
• h5_objref.py (c,f90)
• Creates a dataset with object references
1.
2.
3.
4.

group = f.create_group("G1")
Scalar dataspace
dataset = f.create_dataset("DS2",(), 'i')
# Create object references to a group and a dataset
refs = (group.ref, dataset.ref)

5. ref_type = h5py.h5t.special_dtype(ref=h5py.Reference)
6. dataset_ref = file.create_dataset("DS1", (2,),ref_type)
7. dataset_ref[...] = refs

April 17-19


45

HDF5 Object References (cont.)
• h5_objref.py (c,f90)
• Finding the object a reference points to:
1.
2.
3.
4.
5.
6.

f = h5py.File('objref.h5','r')
dataset_ref = f["DS1"]
print h5py.h5t.check_dtype(ref=dataset_ref.dtype)
refs = dataset_ref[...]
refs_list = list(refs)
for obj in refs_list:
print

April 17-19

f[obj]


46

HDF5 Dataset Region References
• h5_regref.py (c,f90)
• Creates a dataset with region references to each
row in a dataset
1.
2.
3.
4.

refs = (dataset.regionref[0,:],…,dataset.regionref[2,:])
ref_type = h5py.h5t.special_dtype(ref=h5py.RegionReference)
dataset_ref = file.create_dataset("DS1", (3,),ref_type)
dataset_ref[...] = refs

April 17-19


47

HDF5 Dataset Region References (cont.)
• h5_regref.py (c,f90)
• Finding a dataset and a data region pointed by a
region reference
1.
2.
3.
4.
5.
6.

path_name = f[regref].name
print path_name
# Open the dataset using the pathname we just found
data = file[path_name]
# Region reference can be used as a slicing argument!
print data[regref]

April 17-19


48

Hints
• When to use HDF5 object references?
• Instead of an attribute with a lot of data
• Create an attribute of the object reference type and
point to a dataset with the data

• In a dataset to point to related objects in HDF5 file

• When to use HDF5 region references?
• In datasets and attributes to point to a region of
interest
• When accessing the same region many times to
avoid hyperslab selection process

April 17-19


49

Partial I/O

Working with subsets

April 17-19


50

Collect data one way ….
Array of images (3D)

April 17-19


51

Display data another way …

Stitched image (2D array)

April 17-19


52

Data is too big to read….

April 17-19


53

How to Describe a Subset in HDF5?
• Before writing and reading a subset of data
one has to describe it to the HDF5 Library.
• HDF5 APIs and documentation refer to a
subset as a “selection” or “hyperslab
selection”.
• If specified, HDF5 Library will perform I/O on a
selection only and not on all elements of a
dataset.

April 17-19


54

Types of Selections in HDF5
• Two types of selections
• Hyperslab selection
• Regular hyperslab
• Simple hyperslab
• Result of set operations on hyperslabs
(union, difference, …)

• Point selection

• Hyperslab selection is especially important for
doing parallel I/O in HDF5 (See Parallel HDF5
Tutorial)

April 17-19


55

Regular Hyperslab

Collection of regularly spaced equal size blocks

April 17-19


56

Simple Hyperslab

Contiguous subset or sub-array

April 17-19


57

Hyperslab Selection

Result of union operation on three simple hyperslabs

April 17-19


58

Hyperslab Description
• Start - starting location of a hyperslab (1,1)
• Stride - number of elements that separate each
block (3,2)
• Count - number of blocks (2,6)
• Block - block size (2,1)
• Everything is “measured” in number of elements

April 17-19


59

Simple Hyperslab Description
• Two ways to describe a simple hyperslab
• As several blocks
• Stride – (1,1)
• Count – (3,4)
• Block – (1,1)

• As one block
• Stride – (1,1)
• Count – (1,1)
• Block – (3,4)

No performance penalty for
one way or another
April 17-19


60

Writing and Reading a Hyperslab
• Example h5_hype.py(c, f90)
• Creates 8x10 integer dataset and populates with data; writes
a simple hyperslab (3x4) starting at offset (1,2)
• H5Py uses NumPy indexing to specify a hyperslab
• Numpy indexing array[i : j : k]
• i – the starting index; j – the stopping index; k – is the step (≠ 0)

dataset[1:4, 2:6]

offset

April 17-19

count+offset


61

Writing and Reading Simple Hyperslab
dataset[1:4, 2:6] = 5
print "Data after selection is written:"
print dataset[...]
[[1
[1
[1
[1
[1
[1
[1
[1

April 17-19

1
1
1
1
1
1
1
1

1
5
5
5
1
1
1
1

1
5
5
5
1
1
1
1

1
5
5
5
1
1
1
1

2
5
5
5
2
2
2
2

2
2
2
2
2
2
2
2

2
2
2
2
2
2
2
2

2
2
2
2
2
2
2
2

2]
2]
2]
2]
2]
2]
2]
2]]


62

Writing and Reading Regular Hyperslab
space_id = dataset.id.get_space()
space_id.select_hyperslab((1,1), (2,2), stride=(4,4), block=(
2,2))
dataset.id.read(space_id, space_id, data_selected)
print data_selected
Selected data read from file....
[[0
[0
[0
[0
[0
[0
[0
[0
April 17-19

0
1
1
0
0
1
1
0

0
5
5
0
0
1
1
0

0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0

0
5
5
0
0
2
2
0

0
2
2
0
0
2
2
0

0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0

0]
0]
0]
0]
0]
0]
0]
0]]


63

Writing and Reading Point Selection
• Example h5_selecelem.py(c, f90)
• Creates 2 integer datasets and populates with data; writes a
point selection at locations (0,1) and (0, 3)
• H5Py uses NumPy indexing to specify points in array
val = (55,59)
dataset2[0, [1,3]] = val
[[ 1 55
[ 1 1
[ 1 1

April 17-19

1 59]
1 1]
1 1]]


64

Hints
• C and Fortran
• Applications’ memory grows with the number of
open handles.
• Don’t keep dataspace handles open if
unnecessary, e.g., when reading hyperslab in a
loop.
• Make sure that selection in a file has the same
number of elements as selection in memory when
doing partial I/O.

April 17-19


65

Other Features
Storage, Extendibility, Compression

April 17-19


66

Dataset Storage Options
• Compact
• Used for storing small (a few Ks) data

• Contiguous (default)
• Used for accessing contiguous subsets of data

• Chunked
• Data is store in chunks of predefined size
• Used when:
• Appending data
• Compressing data
• Accessing non-contiguous data (e.g., columns)

April 17-19


67

HDF5 Dataset

Metadata

Dataset data

Dataspace
Rank Dimensions
3

Dim_1 = 4
Dim_2 = 5
Dim_3 = 7

Datatype
IEEE 32-bit float

Attributes
Storage info

Time = 32.4

Chunked

Pressure = 987

Compressed

Temp = 56

April 17-19


68

Examples of Data Storage
Compact
Metadata

Raw data

Contiguous

April 17-19


Chunked

69

Extending HDF5 dataset
• Example h5_unlim.py(c,f90)
• Creates a dataset and appends rows and columns
• Dataset has to be chunked
• Chunk sizes do not need to be factors of the dimension sizes
dataset = f.create_dataset('DS1',(4,7),'i',chunks=(3,3),
maxshape=(None, None))
0
0
0
0
0
0
April 17-19

0
0
0
0
0
0

0
0
0
0
0
0

0
0
0
0
0
0

0
0
0
0
0
0

0
0
0
0
0
0

0
0
0
0
0
0


0
0
0
0
0
0

0
0
0
0
0
0
70

Extending HDF5 dataset
• Example h5_unlim.py(c,f90)
dataset.resize((6,7))
dataset[4:6] = 1
dataset.resize((6,10))
dataset[:,7:10] = 2
0
0
0
0
1
1

April 17-19

0
0
0
0
1
1

0
0
0
0
1
1

0
0
0
0
1
1

0
0
0
0
1
1

0
0
0
0
1
1

0
0
0
0
1
1

2
2
2
2
2
2


2
2
2
2
2
2

2
2
2
2
2
2

71

HDF5 compression
•
•
•

Chunking is required for compression and
other filters
HDF5 filters modify data during I/O operations
Compression filters in HDF5
•
•
•
•

April 17-19

Scale + offset (H5Pset_scaleoffset)
N-bit (H5Pset_nbit)
GZIP (deflate) (H5Pset_deflate)
SZIP (H5Pset_szip)


72

HDF5 Third-Party Filters
• Compression methods supported by HDF5
User’s community
http://www.hdfgroup.org/services/contributions.html
•
•
•
•
•

April 17-19

LZF lossless compression (H5Py)
BZIP2 lossless compression (PyTables)
BLOSC lossless compression (PyTables)
LZO lossless compression (PyTables)
MAFISC - Modified LZMA compression filter,
(Multidimensional Adaptive Filtering Improved Scientific
data Compression)


73

Compressing HDF5 dataset
• Example h5_gzip.py(c,f90)
• Creates compressed dataset using GZIP compression
with effort level 9
• Dataset has to be chunked
• Write/read/subset as for contiguous (no special steps are
needed)

dataset =
f.create_dataset('DS1',(32,64),'i',chunks=(4,8),compressi
on='gzip',compression_opts=9)
dataset[…] = data

April 17-19


74

Hints
• Do not make chunk sizes too small (e.g., 1x1)!
• Metadata overhead for each chunk (file space)
• Each chunk is read at once
• Many small reads are inefficient
• Some software (H5Py, netCDF-4) may pick up
chunk size for you; may not be what you need
• Example: Modify h5_gzip.py to use
dataset =
file.create_dataset('DS1',(32,64),'i',compression='gzip
',compression_opts=9)
Run h5dump –p –H gzip.h5 to check chunk size
April 17-19


75

More Information
• More detailed information on chunking can be
found in the “Chunking in HDF5” document at:
http://www.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html

April 17-19


76

Thank You!

April 17-19


77

Acknowledgements
This work was supported by cooperative agreement
number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the author[s] and do not necessarily reflect
the views of the National Aeronautics and Space
Administration.

April 17-19


78

Questions/comments?

April 17-19


79

Advanced HDF5 Features

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Advanced HDF5 Features (20)

More from The HDF-EOS Tools and Information Center (20)

Recently uploaded (20)

Advanced HDF5 Features

Editor's Notes