Icy Moon Surface Simulation and Stereo Depth Estimation for Sampling Autonomy CL24_2268.pdf
- 1. Icy Moon Surface Simulation and Stereo Depth Estimation for Sampling Autonomy Ramchander Bhaskara1, Georgios Georgakis2, Jeremy Nash2, MarissaCameron2, Joseph Bowkett2, Adnan Ansar2, Manoranjan Majji1,Paul Backes2 1Texas A&M University | 2Jet Propulsion Laboratory
- 2. Motivation Engineering representation for icy-moon surface exploration at sampling scale. 1. System design: Spectral understanding for surface autonomy a. Imaging sensor designand selection b. Computer vision algorithms 2. Analysis: Most challenging environments for visual perception Will legacy vision pipelines be sufficient for perception? 2
- 3. Topics 1. Review: Challenges in icy-moon surface simulations 2. Software: Graphical Utility for Icy-Moon Surface Simulations (GUISS) 3. Analysis: Stereo Depth Estimation • Stereo matching evaluation under challengingvisual hypotheses 4. Conclusion & ongoing work: Are simulations enough? 3
- 4. 1. Review Icy world study | Objectives | Challenges | Literature 4
- 5. 1. Review – the why. 5 • Hydrothermal activity – Enceladus/Europa • Europa Clipper • Jupiter Icy Moons Explorer (JUICE) • Decadal survey: Enceladus Orbilander Enceladus plume - Cassini Source: JPL/NASA SAR image mosaic Source: JPL/NASA
- 6. 1. Review – the objectives. 6 Enceladus plume - Cassini Source: JPL/NASA SAR image mosaic Source: JPL/NASA • Sampling of plume materials • Hard engineeringproblem • Perception system goals: • Recover topography(DEM) and texture for site selection • Fault identification • Localization • Excavation tracking Perception systemdevelopment for sampling autonomy
- 7. 1. Review – the challenges. • Structural diversity • Plain, dominated by ice blocks • Surface disruptionsand faulting • Largely unknownat high resolution • Spectral diversity • High albedo > 0.8 (Enceladus) • Backscattering • Crystalline and amorphousice • Freshsnow, plumes, and salts • Subsurface water • Ice thickness: 3-30 kms 7 4m/pixel Enceladus’South pole terrain (Cassini, JPL/Caltech) Structural Mapping of Enceladus (CassiniISS, JPL/NASA)
- 8. 1. Review – Literature. • Scientific rendering for planetary surface simulations 8 Simulator Features SurRender (Airbus) large scalesimulations. NaRPA (Texas A&M) Limited to Lambertian models. SurRender (Airbus) large scalesimulations, validation of radiometric and sensor models. Shader limitations. PANGU (ESA) Standard in vision based navigation.Limited availability. SISPO(Univ. of Tartu) Image aberrations and environment models. OceanWATERS (NASA) No systematictreatmentof reflectancemodels. DUST (NASA) Lunar simulations at scalewith accurate lighting, DTMs integrated. DARTS (JPL) SAELSim– focus is on lander dynamics.
- 9. 2. Rendering system Overview | Terrain | Texture | Lighting | Sensor models 9
- 10. 2. Graphical Utility for Icy-Moon Surface Simulations (GUISS) Overview • Python application using Blender Cycles • Easy to adopt and reconfigure • Open shading language (OSL) • Building blocks • Terrain modeling • Texture modeling • Lighting • Sensor 10 GUISS rendering pipeline Software: https://github.com/nasa-jpl/guiss
- 11. 2. GUISS – Terrain modeling. 11 • Diverse observation geometries – procedural methods • FractionalBrownian motion • Import DEMs (Earth analogues) • Custom meshes (glaciers) • Features: rock distribution Terrain modeling Source: Crow-Willard et al.|Enceladus structural diversity
- 12. 2. GUISS – Terrain modeling. 12 • Procedural elevation models • FractionalBrownian motion • Smooth randomness throughnoise functionsto construct self-similarity. Primitives Multi-fractal Hetero-terrain Hybrid Voronoi Blender: Mesh with noise variations Gaea: Perlin noise example
- 13. 2. GUISS – Terrain modeling. 13 Scenereconstructions from Europa lander field testingcampaign Matanuska Glacier, AK. Rocky and ridged terrainsfrom Gaea software. Procedural multi-fractal terrains,icy penitente features.
- 14. 2. GUISS – Terrain modeling. 14 Rock distribution • Varying scales and densities • meta data. RGB image Semantic segmentationmask Instance segmentationmask
- 15. 2. GUISS – Texture modeling. 15 • Spectrally challenging environments • High albedo (Europa 0.64, Enceladus 0.81) • Roughness(fresh snow – very low) • Specular • Subsurface scattering • Texture (UV) mapping • Procedural texture displacement. Texture modeling Shader used: Principled Bidirectional ScatteringDistribution Function (PrincipledBSDF)
- 16. 2. GUISS – Texture modeling. 16 • UV mapping • Field campaigns– dense reconstruction • Catalog of albedo, displacement, normal, transmission,roughness maps • UV texture mapping • ConsistentUV scaling: map object coordinatesto UV coordinates • Blending texturedtiles ReconstructedUV texture Albedo and transmission maps: snow and rocks Example renders: UV mapping onto a mesh
- 17. 2. GUISS – Texture modeling. 17 • Procedural texture • Effectto displace the regolithic surface • Variationsthrough changesin surface normals • Proceduraladjustments to surface roughnessand normals • Blender Cycles (Principled BSDF) for realismand subsurfacescattering effects. Representativescene:procedural displacement
- 18. 2. GUISS – Texture modeling. 18 Subsurface scattering Transmission factor
- 19. 2. GUISS - Lighting. • Illumination model • Directand indirectsources • Scaled models • Angular diameter • Relative distances • Irradiance and observation geometry: NAIF. 19 Illumination model NAIF: Navigationand Ancillary Information Facility Representative terrain:Sun elevation angles
- 20. 2. GUISS - Sensor. 20 • Camera • Blender supported capabilities • Motion blur • Lens type and distortion • Panoramic and stereoscopic • Noise and denoising filters • Output • Stereo image pair • Groundtruth depth • Segmentation passes. Stereoimage pair Semantic mask Instance mask Depth map
- 21. 3. Stereo depth estimation analysis Sampling autonomy | Baselines | Datasets | Evaluation | Results 21
- 22. 3. Depth estimation – Baselines. 22 • StereoBM1 • Block Matching • # Disparities • Sum of Absolute Differences(SAD) • JPLV2 • Used on Mars Exploration Rover • DSMNet3 • Deep-learningbased • Improved generalizationto new visual domains. • IGEV4 • State-of-the-art. 1Scharstein and R. Szeliski,“A taxonomyandevaluation of dense two-frame stereo correspondence algorithms,”International journal of computer vision, vol. 47,pp. 7–42,2002. 2S.B.Goldberg, M.W. Maimone,andL. Matthies,“Stereo vision androver navigationsoftwarefor planetary exploration,”in Proceedings, IEEE aerospace conference, vol. 5.IEEE, 2002,pp.5–5. 3Zhang,X.Qi, R.Yang,V. Prisacariu,B. Wah,and P. Torr,“Domain-invariant stereo matchingnetworks,”in Computer Vision–ECCV 2020:16thEuropean Conference, Glasgow,UK, August 23–28,2020,Proceedings,Part II 16. Springer, 2020,pp.420–439. 4G. Xu, X. Wang, X.Ding, andX. Yang,“Iterative geometry encoding volume for stereo matching,”in Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition,2023,pp. 21 91921 928.
- 23. 3. Depth estimation – Datasets. 23 • Procedural terrain variation • Planar to highly ruggedterrains • Texture variation • UV maps and albedo maps • BSDF parameters • Albedo, specularity, roughness, transmission,subsurfacefactor. • Lighting variations • Sun elevation angles • Reconstructions Total Images Reconstructe d Simulated 5020 3060 1960
- 24. 3. Depth estimation – Results. 24 • Qualitativeevaluation • Deep learning methods outperformclassicalmethods • Trained on visually dissimilar domains • JPLV - comparable errorsfor real reconstructedscenes.
- 25. 3. Depth estimation – Metrics. 26 • StereoBM: 87 cm & 10.1% gap on L1 and DoD metrics. • 2.3% DOD result of IGEV Performance evaluation on the datasets. • L1 Errorof DSMNet and IGEV are 0.29 and 0.26 – errorsare still significant • IGEV’s 0.03 si-RMSE (scale-invariant).
- 26. 3. Depth estimation – Metrics. 27 • Varying values of Albedo • StereoBM and JPLV degrade • Deep learning models exhibit robustnessto high brightness. Metricswith varying values of albedo.
- 27. 3. Analysis – Runtimes. 28 Rendering runtime • Reconstructed scenes: 3 seconds (Intel i7 CPU) per stereopair and truth (640 x 480) Synthetic scenes:1 minute. Stereo depth estimation runtime • Intel i7 CPU, NVIDIA RTX 3080 GPU (resolution:640 x 480).
- 28. 4. Conclusion and ongoing work Limitations | Development 29
- 29. 4. Conclusion – Limitations. 30 • Fidelity • Renderingsare to be validated. • Photometric model • Hapke type BRDF models. • SPICE Kernel integration • Missionlike conditions • Sensor model • Real sensor(noise models) • Lidar, structured light – renderingand depth estimation Lidar simulation development.
- 30. 4. Conclusion – Limitations. 31 • Fidelity • Renderingsare to be validated. • Photometric model • Hapke type BRDF models. • SPICE Kernel integration • Missionlike conditions • Sensor model • Real sensor(noise models) • Lidar, structured light – renderingand depth estimation Lidar simulation development.
- 31. 4. Conclusion – Limitations. 32 • Fidelity • Renderingsare to be validated. • Photometric model • Hapke type BRDF models. • SPICE Kernel integration • Missionlike conditions • Sensor model • Real sensor(noise models) • Lidar, structured light – renderingand depth estimation Hapke BRDF PrincipledBSDF
- 32. 33 https://github.com/nasa-jpl/guiss Questions We acknowledge the inputs of JPL technologistsIssa Nesnas,Yang Cheng, Asher Elmquist, Travis Driver, Gregory Griffin, Spencer Diehl, Ishan Mishra, Ashish Goel, Anup Katake,Reg Willson, Micheal Swan, Deegan Atha, Daniel Moreno, and Samuel Howell.
- 33. 34 Backup slides Noise scale: 0.5 Noise scale: 3