HDF4 Mapping Project Update
Download as PPTX, PDF0 likes1,733 views
The document discusses the HDF4 Mapping Project which aims to ensure long-term access to Earth Observing System (EOS) data stored in HDF4 files. It provides an overview of the project scope, including developing a proof of concept prototype and production quality mapping tools. It also describes verification studies conducted with NASA data centers to identify requirements for verifying correctness of HDF4 file content maps produced by the mapping tools. The project aims to generate content maps for HDF4 files containing valuable EOS data before the HDF4 library and tools are no longer maintained.
HDF4 Mapping Project Update
- 1. The HDF Group HDF4 Mapping Project Update www.hdfgroup.org/projects/h4map Ruth Aydt (aydt@hdfgroup.org) The HDF Group The 15thHDF and HDF-EOS Workshop April 17-19, 2012 Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 1 www.hdfgroup.org
- 2. Project Motivation HDF4 file DVD HDFView Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 2 HDF4 Library www.hdfgroup.org
- 3. Project Purpose Ensure long-term access to EOS data stored in HDF4 files. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 3 www.hdfgroup.org
- 4. Project Scope April 2012 Time HDF4 Library HDF4 Files with EOS Data produced HDF4 Files with EOS Data valuable to community Concern Idea HDF4 Mapping Project Scope Proof of Concept Prototype Develop Support Product Verification Requirements Study ? Verification Implementation HDF4 File Content Maps Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 4 www.hdfgroup.org
- 5. Concern – Workshop VIII (2004) “HDF and HDF EOS: Implications for Long-Term Archiving and Data Access” - Ruth Duerr, NSIDC Slide Notes: “Without human readability you are locked into having to maintain the read software forever!” Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 5 www.hdfgroup.org
- 6. Idea – Workshop X (2006) “Leveraging HDF Utilities” - Chris Lynnes, GES-DISC Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 6 www.hdfgroup.org
- 7. HDF4 File Contents – User View Objects & Relationships Object Data User Metadata Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 7 www.hdfgroup.org
- 8. HDF4 File Contents – Format View variable name = variable_name rank type storagetype 1 Vgroup name = variable_name class = Var0.0 1 1 Object Data 1 1 1 0...1 SD 1 SDD 1 0...1 data 0…* byte order, chunked storage, compression, … 1 1 0...1 NT 1 1 1 1 1 1 NDG 0…* Vdata name = attribute_name class = Attr0.0 attribute name = attribute_name Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 8 www.hdfgroup.org
- 9. Proof of Concept (8/07- 7/08) • Categorize HDF4 data held by NASA • Build a prototype HDF4 File bytestreams Map Writer linked with HDF4 library request Reader HDF4 File Content Map (XML) Objects & Relationships; User Metadata; Object Data retrieval & reconstruction information 2 independent readers in C and Perl Object Data Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 9 www.hdfgroup.org
- 10. Develop Product (11/09 - 7/11) Tasks: A. Investigate integration of mapping schema with existing standards B. Determine HDF-EOS 2 requirements C. Redesign and expand the XML schema D. Implement production quality map writer E. Develop demo map reader F. Deploy tools at select NASA data centers For preservation, we must get it right while the HDF4 library, tools, documentation, and expertise are around. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 10 www.hdfgroup.org
- 11. Develop Product (Tasks C & D) C: HDF4 File Content Maps Have enough information to stand alone • Described by schema D: Production Quality Map Writer • Read HDF4 file and create Map • Command-line options fine-tune behavior HDF4 Library • New functions added to facilitate map creation Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 11 www.hdfgroup.org
- 12. Surprise! • Expected hardest part to be support for retrieval and reconstruction of object data. • In fact, making sure all user-created HDF4 objects were found and represented correctly was a bigger challenge. • Existing tools didn’t always report same user-level information. • “Correctness” can be subject to interpretation – not always able to know intent of file creator. Image from publications.usa.gov Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 12 www.hdfgroup.org
- 13. Project Actions in Response User View • Map from top down andbottom up • Watch for extra parts • “Over include” in map if any doubt (e.g., 2 palettes for 1 raster) Format View • Improve HDF4 library, tools, and documentation to address ambiguities Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 13 www.hdfgroup.org
- 14. HDF4 File Content Map Select object data values Information needed Represents HDF4 included to help reader to access and Objects and program verify binary interpret object data dataRelationships handled properly in HDF4 file Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 14 www.hdfgroup.org
- 15. E: Develop Demo Reader Developed by student at NSIDC Only given Content Maps • Written in Python • Reader extracts object data from HDF4 file • Output in ASCII (csv) or binary (numpy) • Compares extracted data to values for verification in Content Map Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 15 www.hdfgroup.org
- 16. Releases & Support Date Version Comments July 2011 1.0.0 schema 1.0.0 writer First official release http://www.hdfgroup.org/projects/h4map Sept 2011 1.0.1 writer Minorbug fixes Nov 2011 1.0.1 schema 1.0.2 writer Robustly handle empty SDS March 2012 May 2012 (planned) ? Apr. 17-19, 2012 ECS Release 8.1 1.0.3 writer Minor bug fixes Support 2 palettes with same reference number HDF/HDF-EOS Workshop XV 17 www.hdfgroup.org
- 17. HDF4 File Content Maps Content Map generation at GES-DISC • Datasets mapped • TOVS Pathfinder For example: ftp://disc1.gsfc.nasa.gov/data/s4pa/tovs/TOVSADNG/1986/330/ • MERRA Model Output • In progress • TRMM • AIRS Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 18 www.hdfgroup.org
- 18. ECS Release 8.1 – March 2012 “Raytheon EED deployed the HDF4 File Content Maps capability as part of ECS Release 8.1. This capability wraps the Content Map Writer in the ECS Map Generation Server. ECS DAACs can choose whether or not to enable map generation in operations. With workload spec testing, seeing 2-3 maps/second under load and 10-15 on unloaded system” -- Evelyn Nakamura, Raytheon “We installed our new big ECS software release which included the code for creating maps. The installers set it up to create maps (not in operations mode) for MOD10A1 and it produced 20 or 30 thousand. We haven't had a chance to look at them yet.” -- Doug Fowler, NSIDC Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 19 www.hdfgroup.org
- 19. Verification* Study (1/12 - 4/12) “Work with DAAC personnel to identify requirements that would produce appropriate and efficient methods of verifying, concurrent with operation activities, correctness of the HDF4 maps that are produced with the ECS 8.1 capability.” * The terms Verification and Validation are used interchangeably. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 20 www.hdfgroup.org
- 20. Verification Study Activities Webinars with ASDC, LPDAAC, NSIDC, Raytheon • Provide background on Mapping Project • Gather input on requirements and concerns • Collect sample datasets and generate Content Maps Exposed 3 bugs: 1 in HDF4 library & 2 in Map Writer; Fixed. • Discuss possible approaches • Seek guidance from NASA on expectations regarding Map creation timeline and verification responsibilities Prototype possible approaches • Demonstrate functionality and assess feasibility Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 21 www.hdfgroup.org
- 21. Verification Study Findings (1) • Automate verification as much as possible. • Focus verification at the ESDT version level. • No definitive specification for user-level objects expected in a given HDF4 file. • Scientists look at visualizations, not directly at data. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 22 www.hdfgroup.org
- 22. Verification Study Findings (2) • Every DAAC is different • Flexibility in deciding when to generate Maps • May need involvement of science teams to confirm correctness • Content Maps should be produced near end of mission, or sooner if users want them. • AMSR-E identified • NSIDC involved with Mapping project from the start and comfortable with verification using demo reader Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 23 www.hdfgroup.org
- 23. Verification Study Findings (3) • Interest in web-based tools is growing. • XSLT stylesheets • DAAC representatives are very concerned about long-term access to data. • This is beyond the scope of the study • But, something to keep in mind when considering different approaches Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 24 www.hdfgroup.org
- 24. Verification Dilemma Translator to DVD Reader Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 25 www.hdfgroup.org
- 25. Possible Approach DVD DVD Creator DVD Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 26 www.hdfgroup.org
- 26. Applied to Content Maps HDF4 File Content Map (XML) HDF4 File request bytestreams HDF4 Reader Retranslator Objects & Relationships; Relationships; User Metadata; Metadata; Object Data retrieval & Object Data retrieval & reconstruction information reconstruction information Object Data HDF4 File Replace this… with this… Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 27 www.hdfgroup.org
- 27. Verification Recommendations (1) • Check h4mapwriter errors • Run xmllint • Check for well-formed XML • Validate Map conforms to schema These checks are possible now Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 28 www.hdfgroup.org
- 28. Verification Recommendations (2) • Develop content map checker to check • • • • Filesize and checksum Object data values Values for verification Attribute values in Map What people expect to be enough Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 29 www.hdfgroup.org
- 29. Verification Recommendations (3) • Develop retranslatorto create new HDF4 file • Allows use of familiar tools (GrADS, IDL, HDFview, hdiff, …) • If new file is not equivalent to original (from user perspective), investigate ASAP. Needed since no definitive source of correctness for original HDF4 files. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 30 www.hdfgroup.org
- 30. Verification Recommendations (4) • Build content map checker and retranslatoron common modular infrastructure. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 31 www.hdfgroup.org
- 31. Not just for Preservation! “I find the HDF Map writer and reader very useful when I am in the discovery phase of new projects using HDF4 datasets. • They enable me to analyze the full structure of CERES hdf4 datasets and ensure HDF Attributes from the archived HDF4 files are preserved in subsetted files. • I am building a capability to subset MOPITT HDF4 data and am using them to help validate SDS data arrays over 4 dimensions. • A team of consultants is working with ASDC on an experimental semantic database implemented on a 'grand challenge' scale. They are interested in using CERES datasets, but are unfamiliar with HDF. They are using the HDF4 map application to analyze the structure of proposed CERES datasets and to help extract metadata and data from target files.” --- Walt Baskin, ASDC Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 32 www.hdfgroup.org
- 32. Presentation “Take Away” HDF4 Content Maps are the best thing since sliced bread! More seriously … • • Content Maps can be created now and you may find them useful Ask questions and report problems We want to know about issues ASAP • Feedback regarding proposed Verification approach very welcome Project report / recommendations due next week Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 33 www.hdfgroup.org
- 33. Project Contributors • The HDF Group • Ruth Aydt, Peter Cao, Jo Eads, Mike Folk, Joe Lee, Elena Pourmal, Binh-Minh Ribler, Kent Yang, and others • NASA / DAACs • Jeanne Behnke, Dan Marinelli, H. K. "Rama" Ramapriyan • ASDC: Walt Baskin, Greg Cates, Gerald Lemay, Lindsay Parker, Steve Protack • GES-DISC: Guang-Dih Lei, Chris Lynnes • LP DAAC: Matt Martens, BhaskarRamachandran, Jody Rundell, Jim Vermeer • NSIDC: Jonathan Crider, Ruth Duerr, Doug Fowler, Luis Lopez • Raytheon • Evelyn Nakamura, Lou Swentek, Abe Taaheri Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 34 www.hdfgroup.org
- 34. Acknowledgements This work was supported by Subcontract number 114820 under RaytheonContract number NNG10HP02C, funded by the National Aeronautics andSpace Administration (NASA) and by cooperative agreement numberNNX08AO77A from the NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authorsand do not necessarily reflect the views of Raytheon or the NationalAeronautics and Space Administration. Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 35 www.hdfgroup.org
- 35. The HDF Group Questions/comments? Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 36 www.hdfgroup.org
Editor's Notes
- #19: TOVS Pathfinder: http://mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=project&project=TOVSMERRA Model Output:mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=project&project=MERRATo find the map files, you go down all the way to the granule level, then copy the FTP link and take off the file part, e.g.,:ftp://disc1.gsfc.nasa.gov/data/s4pa/tovs/TOVSADNG/1986/330/Thanks to Chris Lynnes for the info & links.
- #20: Maps can be generated. Because of concerns that they can’t be verified in an automatic, scalable way, don’t have to be turned on. Verification Study.
- #21: With the ability to generate content maps, DAACs wanted to know how they should verify that dataset files are adequately described… In many cases they were not responsible for creating the files or for understanding the content in them… they typically just look at checksums, filesizes, before distributing. In part because of our surprise in the product phase, we felt it would be best to discuss some of the uncertainties related to verification – why just comparing the values in the object data isn’t enough and how the uncertainty regarding creator intent (in some cases) could be addressed.
- #22: Here’s a high-level rundown of the activities that have gone on during the project. DAAC personnel have been very responsive to questions and made room in their schedules to meet on fairly short notice.
- #24: A summary of the findings. Details are in meeting minutes.
- #25: Will a “Map Reader” replace the HDF4 library as the way to access data at some point in the future?How will a “Map Reader” or other utilities be supported?