Enhancing usability and utility of USGS 3D Elevation Program (3DEP) lidar data and web services with Jupyter Notebooks
0 likes25 views
The document discusses enhancements to the USGS 3D Elevation Program (3DEP) through customizable Python-based Jupyter notebooks, allowing users to access, process, and visualize LIDAR data easily. It highlights the availability of over 1,800 datasets and more than 43 trillion LIDAR points, with ongoing data acquisition. The project aims to improve usability and accessibility of 3DEP data using cloud resources and interactive workflows, benefiting various applications within the geospatial community.
Enhancing usability and utility of USGS 3D Elevation Program (3DEP) lidar data and web services with Jupyter Notebooks
- 1. https://registry.opendata.aws/usgs-lidar/ Funding for this ongoiing work was provided by the Community for Data Integration (CDI) from the project grant:“Enhancing usability of 3DEP data and web services with Jupyter notebooks • User installs Python modules and runs Jupyter Notebook(s) on personal workstation. • Performance and job size capabilities depend on local hardware. • User runs Jupyter Notebook in Google Colaboratory environment on Google’s Cloud Platform. • No local Python installs or processing required. • Allocations of 12GB RAM and Google’s high performance CPU/GPUs • Products directly downloaded to and accessed from the user’s Google Drive Enhancing usability and utility of USGS 3D Elevation Program (3DEP) lidar data and web services with Jupyter Notebooks Cole Speed1,3 , Matthew Beckley1 , Christopher Crosby1 , Viswanath Nandigam2 !"#$%&"&'()"*+ Contact: cole.speed@beg.utexas.edu 1 UNAVCO Inc.; San Diego Supercomputer Center, UC San Diego; 3 Department of Geological Sciences Jackson School of Geosciences, University of Texas at Austin 600 km • 3DEP data are available as an Amazon Web Services Public Dataset in Entwine Point Tile (EPT) format for public use. • Currently, >1800 datasets, >43 trillion lidar points, > 6 million km2 coverage with data acquistion ongoing. • Project Goal: Develop accessible and usable workflows for accessing, processing, and visualizing 3DEP data via customizable Python-based Jupyter Notebooks and cloud resources. The 3D Elevation Program (3DEP) ipyleaflet Project Overview • The 3D Elevation Program, managed by the US Geological Survey, is acquiring quality level 2 or better light detecting and ranging (lidar) data over the conterminous United States, Hawaii, and US Territories to meet the growing need for high-resolution 3-D representations of Earth's surface, vegetation, and constructed features. • OpenTopography, supported by the USGS Community for Data Integration (CDI) program, is developing well-documented and customizable Jupyter Notebook-based Python workflows for programmatically accessing, processing, and visualizing 3DEP data products for a variety of use-cases geared toward USGS applications and for users of point cloud data across the geospatial community. Example Use-Cases of Jupyter Notebook 3DEP Workflows z Distribution, Installation, and Execution Python Workflows for Accessing, Processing, and Visualizing 3DEP Data Lidar Point Cloud CHM DTM by USGS Watershed (12 Digit HUC Service) Cutsom DTM/DSMs for USGS 7.5 x 7.5’ Quadragles Canopy Height Model for User-defined AOI https://github.com/cmspeed/OT_3DEP_Workflows Topographic Differencing of 3DEP Datasets We are seeking suggestions for specific use-cases! USGS 3DEP 7.5 x 7.5’ Quadrangle DTM DTM Differenced DSM USGS 3DEP 12-Digit USGS WBD USGS 3DEP User-defined AOI USGS 3DEP (2013) USGS 3DEP (2011) User-defined AOI Lidar Point Cloud 5 km 5 km 5 km Colorize Point Clouds by Corresponding NAIP Imagery NAIP Image Service Correspoinding 3DEP Datset Colorized point clouds Text about querying 3DEP Colorized point clouds 1 km USGS 3DEP User-defined AOI 1 km 1 km 1 km 5 km 3 km DSM DTM Canopy Height (m) 0 5 2 km USGS Ground Class - Lidar Points 5 km USGS Ground Class - Lidar Points DTM 5 km Get 3DEP Polygons via API Construct PDAL Pipelines Execute the PDAL Pipelines Define AOI on Interactive Map API request to get up-to-date 3DEP polygons from USGS lidar repository (https://github.com/hobu/usgs-lidar) Extract names, geometries, and point counts for each dataset AOI subset of 3DEP data is downloaded from aws usgs-lidar Entwine Point Tile (EPT) bucket Processing stages are applied Products saved locally or to Google Drive Workflow leverages PDAL (Point Data Abstraction Library) Customizable PDAL pipelines: specify resolution and processing steps (e.g., filtering, reclassifying, colorizing, gridding) iPyLeaflet interactive maps / draw methods / shapefiles for user AOI Estimate point density for AOI iPywidgets promote customization and ease-of-use Option 2: Install and Execute on Google Colaboratory Option 1: Local Installation and Execution QR Code to Github Repository 2013 - DTM 2011 - DTM Topographic Difference (2013-2011) (m) -0.5 0.5 (m) 2 km 1 km 3DEP Point Cloud and Digital Terrain Model (DTM) Diamond Head Volcano, HI Colorized 3DEP Point cloud