Proposal for adding Named Dimensions to HDF5 Arrays
Download as PPT, PDF1 like557 views
The proposal discusses adding named dimensions to HDF5 arrays to better support metadata for data produced by the Ozone Mapping and Profiler Suite (OMPS) using HDF5. The method aims to retain the flexibility of HDF5's group hierarchy while implementing this feature, which could enhance data representation without altering the base library. The presentation outlines potential challenges and benefits, including inheritance of named dimensions and alternative representations for compound data types.
Proposal for adding Named Dimensions to HDF5 Arrays
- 1. Proposal for adding Named Dimensions to HDF5 Arrays Daniel Kahn Science Systems and Applications, Inc. HDF/HDF-EOS Meeting Oct 15-17th, Aurora CO HDF/HDF-EOS Meeting Oct 15-
- 2. Motivation: The Ozone Mapping and Profiler Suite (OMPS) data produced by the O3 PEATE will be using HDF5, but not HDF-EOS5. Features of HDF-EOS5 needed by the PEATE are easily represented in HDF5, with the exception of named array dimensions. This talk describes a proposed method to add this functionality to HDF5 and also shows how named dimensions could be used to describe other types of useful data structures. HDF/HDF-EOS Meeting Oct 15-
- 3. Prior Art: HDF-EOS5 Swaths An HDF-EOS5 Swath consists of a collection of arrays, generally as a result of a satellite remote sensing measurements (i.e., an orbit's worth of data). Rectangular array of ozone data projected on Earth. HDF/HDF-EOS Meeting Oct 15-
- 4. The HDF-EOS5 Swath consists of a 1) list of named dimension (Name-Value pairs) metadata. 2) set of arrays whose axes are defined by the named dimensions. HDF-EOS5 achieves this via a simple API, kept simple by restricting the user from defining HDF5 group hierarchies. File Root Dim Metadata Swath 1 Dataset1 Dataset2 Swath 2 etc. HDF-EOS5 Swath Cartoon Diagram HDF/HDF-EOS Meeting Oct 15- Dim Metadata
- 5. Goal: Offer a corresponding facility under HDF5 1) without sacrificing the ability to define arbitrary hierarchies and 2) using publicly defined attributes associated directly with the HDF5 objects they help to describe. HDF/HDF-EOS Meeting Oct 15-
- 6. Named Dimensions vs Numerical Dimensions HDF5 uses numerical dimensions: hid_t H5Screate_simple(int rank, const hsize_t * dims, const hsize_t * maxdims) dims is an array of numbers. A named dimension is a Name-Value pair used to define arrays. The equivalent statement using named dimensions would look like: H5ext_create_simple(rank, char **DimNames) HDF/HDF-EOS Meeting Oct 15-
- 7. A challenge in defining named extents in HDF5 is that we need some idea of the “scope” of the names within the HDF5 Group hierarchy. G G G Attributes Should these groups see the Extent Names in G*? G G* NameValue1 NameValue2 etc. Presumably not, but some kind of inheritance would be nice. HDF/HDF-EOS Meeting Oct 15-
- 8. We can define inheritance in which datasets in G* could use Named Dims from G* and also from extents inherited from G* parent, G+, and so on up the hierarchy. G Attributes G G+ NameValue3 NameValue4 etc. Attributes G HDF/HDF-EOS Meeting Oct 15- G* NameValue1 NameValue2 etc.
- 9. What would the code look like? Group = H5Gcreate(“/”,”G1”) H5ext_CreateNamedExtent(Group,“Name1”, Value) etc. char **Names = {“Name1”,”Name2”,”Name3”} ret= H5ext_CreateDataSet(Group, rank, Names, ArrayPointer) This searches up the hierarchy for names. if(ret == Error) then Dimension Name not found in hierarchy. Normal HDF5 writing routines. (close, close, close, etc.) HDF/HDF-EOS Meeting Oct 15- (next, side benefits)
- 10. Additional Benefits: Dimension Scale-like capability built in… Index Map contains indexes of DimName2 HDF/HDF-EOS Meeting Oct 15- DimName1 DimName2 …and the Named Dimension also open the possibility to have scales for incommensurate mappings. Data Scale Index Map DimName1 Scale DimName1 DimName1 Data
- 11. The Named dimension approach also appears to be isomorphic to compound datatypes. Three value compound datatype Field 1 Field 2 Field 3 One array of compound type DimName1 Measurement Group Three ordinary arrays of basic type They seem equivalent but.... HDF/HDF-EOS Meeting Oct 15- Field 2 Field 3 DimName1 DimName1 Rows Field 1 DimName1 Rows
- 12. With the Named Dimension approach fields can be added or deleted trivially and the field elements themselves can easily have other dimensions. Field 4 + DimName1 Field 3 DimName1 Field 2 DimName1 DimName1 Rows Field 1 Di Na m The target language can read/write the data without any dependence on the C language implementation of HDF/HDF-EOS Meeting Oct 15structures. e2 m
- 13. Implementation Thoughts: Most likely implementation would be to publicly define HDF5 attributes to represent the metadata and write C interface to simplify programing. Potential Pitfalls: Order of Dimension Names is language dependent. Need bi-directional parent-child relationships. others? Ideas or comments are welcome and needed, in particular on any gaps or problems with the proposed design. HDF/HDF-EOS Meeting Oct 15-
- 14. Conclusion: The proposed method, called “Named Dimensions”, is expected to provide equivalent data representation to the named dimensions of HDF-EOS5 in vanilla HDF5. In addition, other benefits may be realized, including full use of the HDF5 group hierarchy, a flexible Dimension Scale-like capability and an alternative to representing data in compound data types without depending on C style structures. The Named Dimensions facility needs to be implemented on top of HDF5 with no changes to the base library. (i.e., No PEATE branch of HDF5) using publicly defined attributes. Acknowledgment: This work was carried out under NASA contract NNG06HX18C, task 614.5-01-06 HDF/HDF-EOS Meeting Oct 15-
- 15. More Detailed Implementation Thoughts: Group Attributes Group DimName_001: DimName_002: DimName_003: DimExtent_001: DimExtent_002: DimExtent_003: “Length” “Height” “Width” 40 100 16 DataSet Attributes DimNames: {“Width”,”Height”,”Length”} DataSet 40 100 16 HDF/HDF-EOS Meeting Oct 15-