Hyrax: Serving Data from S3
Download as PPTX, PDF0 likes185 views
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Hyrax: Serving Data from S3
- 1. ESIP-0722 JG Hyrax: Serving Data from S3 Summer ESIP 2022 This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001. This document does not contain technology or Technical Data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. James Gallagher Software Engineer/NASA EED-3 contractor jgallagher@opendap.org
- 2. ESIP-0722 JG 2 • Serve existing files that are stored on S3* • There’s no need to alter the files – just copy them to S3 • Works for HDF5** and netCDF4*** What you can do * Simple Storage Service ** Hierarchical Data Format, version 5 *** Network Common Data Format, version 4
- 3. ESIP-0722 JG 3 • Where the server will run • Where you will store the ancillary metadata files the server needs – These files are not the big data files – They provide a road map to the interior structure of those data files • The URLs* of the data files you want to serve What you need to know * Universal Resource Locators
- 4. ESIP-0722 JG 4 • Use the command line tool (get_dmrpp) to build a DMR++* file • Use get_dmrpp’s default setting and see if the way it represents your data is acceptable (try it and see) • Customize the configuration if needed • Write a script to process your collection of files How to make the ancillary files *Dataset Metadata Response Plus Plus
- 5. ESIP-0722 JG 5 • Get the Hyrax server Docker container • Start the container – it contains both the Hyrax data server and the get_dmrpp command • Run get_dmrpp inside the container – a recipe follows To run get_dmrpp
- 6. ESIP-0722 JG 6 Go to the directory where you want to store the ancillary information Start the docker container that has the Hyrax server and the get_dmrpp tool docker run --hostname hyrax --publish 8080:8080 --volume $(pwd):/usr/share/hyrax --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --name hyrax opendap/hyrax:snapshot If your computer uses the new Apple Silicon processor (M1), add --platform linux/amd64 The get_dmrpp Recipe
- 7. ESIP-0722 JG 7 Run any command in the Docker container docker exec –interactive --tty hyrax /bin/bash Run get_dmrpp in the Docker container docker exec –interactive --tty hyrax /bin/bash -c “cd /usr/share/hyrax; get_dmrpp …” Where get_dmrpp -b . -u https://url.for.your/data/file.nc -o file.nc.dmrpp s3://bucket/objectname get_dmrpp, continued
- 8. ESIP-0722 JG 8 Run any command in the Docker container docker exec –interactive --tty hyrax /bin/bash Run get_dmrpp in the Docker container get_dmrpp -b . -u https://cloudydap.s3.amazonaws.com/samples/1A.GPM.GMI.COUNT2014v3.20160105- S230545-E003816.010538.V03B.h5 -o 1A.GPM.GMI.COUNT2014v3.20160105-S230545-E003816.010538.V03B.h5.dmrpp s3://cloudydap/samples/1A.GPM.GMI.COUNT2014v3.20160105-S230545-E003816.010538.V03B.h5 This will build a DMR++ document for the named granule in cloudydap S3 bucket. The DMR++ will use the URL https://cloudydap.s3... to read data values. get_dmrpp, example
- 9. ESIP-0722 JG 9 • Look at the default output and decide how it should change • Look at the HDF5 Handler documentation for Hyrax – the same options can be used with get_dmrpp • Put those optional ‘keys’ in a file and pass name of that file to get_dmrpp using the command’s -s option How to customize the DMR++
- 10. ESIP-0722 JG 10 • How to build & deploy DMR++ files for Hyrax • Customization options for HDF5 • If you’d like more information or a demonstration, see me after the session More information Some useful Docker commands docker rm -f $(docker ps -aq) # remove all containers docker rmi -f $(docker images -q) # remove all images
- 11. ESIP-0722 JG 11 This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001.