“A Computer Vision System for Autonomous Satellite Maneuvering,” a Presentation from SCOUT Space
0 likes13 views
Scout Space Inc. is developing advanced perception systems for autonomous satellites to enhance their navigation and debris avoidance capabilities. The document outlines their approach to pose estimation, competition results, and lessons learned, emphasizing the importance of synthetic data generation and evaluation for improving system reliability and performance in real-space conditions. Future work includes flight experiments and further refinement of machine learning pipelines to enhance pose estimation for various spacecraft geometries.
“A Computer Vision System for Autonomous Satellite Maneuvering,” a Presentation from SCOUT Space
- 1. Developing a Computer Vision System for Autonomous Satellite Maneuvering Andrew Harris, PhD Senior Systems Engineer SCOUT Space Inc.
- 2. 2 What We’re Talking about Here 1. SCOUT’s pose estimation approach and competition performance 2. How we did and lessons learned 3. Pose estimation demo 4. Future work + challenges for the field © 2023 SCOUT Inc
- 3. • SCOUT is developing perception systems to enable the next- generation of autonomous satellites to avoid debris and keep space safe • Two major domains: • Close range, proximity operations • Long-range, space domain awareness SCOUT: Perception for Spacecraft 3 © 2023 SCOUT Inc 40% 20% 15% 10% 8% 7% Chart Title Category 1 Category 2 Category 3 Category 4 Category 5 Category 6
- 4. Challenges: • Extremely data-limited • Sensitive to safety, correctness issues • Relatively compute constrained Space Domain Vision: Is It Hard, or Just Different? 4 © 2023 SCOUT Inc Prospects: • Low-clutter, typically simple backgrounds • “Knowable” lighting conditions, dynamics • Well-defined shapes (usually)
- 5. SCOUT vs. Traditional Approaches 5 Preprocessing Keypoint Extraction PnP Solver Images Pose Images Traditional Pose Estimation • SCOUT’s proximity operations navigation solution leverages ML-driven pose estimation • Many tradeoffs vs. ‘traditional’ approach • Fewer parameters to tune • Generalizability across lighting domains • Aim is to improve runtime + sensitivity vs traditional approaches ‘s approach ML Pipeline © 2023 SCOUT Inc
- 6. 6 Model Development ● Target modeling ● Synthetic dataset generation ● Pipeline build- out ● Evaluation Verification + Validation ● Evaluation on reserved dataset ● Robustness testing ● MLOps Deployment ● Utilize results in on- board GN&C system pose filters ● Downlink full images, estimated attributes SCOUT: Perception Systems SCOUT Target Database On-Orbit Deployment Field Data New Parameters © 2023 SCOUT Inc
- 7. © 2023 SCOUT Inc SCOUT: The Long Road to Pose 7
- 8. • ESA competition to improve image-driven pose estimation technology • Inspired by the Prisma formation flight mission (right) • SPEED+ dataset: ~10k physical images from SLAB testbed, 60k simulated images from SLAB simulator • Scored based on sum of position and attitude estimation error ESA Kelvin Pose Estimation Challenge: A Motivating Problem 8 © 2023 SCOUT Inc
- 9. ESA Kelvin Pose Estimation Challenge: Closing the Domain Gap 9 sunlamp lightbox © 2023 SCOUT Inc synthetic
- 10. • Blender used as scene generation suite of choice • Naïve / unrefined Earth, S/C parameters • No noise, star background; only resolution challenges First Attempt: Blender 10 © 2023 SCOUT Inc CAD model of a spacecraft (Tango) Photorealistic render of Tango
- 11. “Are the Synthetics Realistic?” 11 11 Photorealistic render of Tango Blender model of Tango from CAD Comparison vs. lightbox image • Render pipeline generally looks good, but is not necessarily realistic • Higher reflectivity than lab Tango • Some component mismatches from real mock-up (see right) • Missing diffuse back-reflection Does it matter, and how do we fix it? © 2023 SCOUT Inc
- 12. • Lighting • Lighting conditions change rapidly on-orbit • Streaking/exposure • Sun angle / glint • Blur sources (focus, motion) • Detector noise • Shot, dark current • Cosmic rays Improving Realism: Things to Consider 12 Right: Photograph of an Iridium flare against star background Left: Long exposure showing multiple suspected cosmic ray hits © 2023 SCOUT Inc
- 13. Stanford Space Rendezvous Laboratory: Augmenting Data with Synthetic Noise 13 © 2023 SCOUT Inc
- 14. Stanford Space Rendezvous Laboratory: Improving Performance with Synthetic Data 14 © 2023 SCOUT Inc
- 15. © 2023 SCOUT Inc Competition Results 15
- 16. Validation 16 © 2023 SCOUT Inc Lightbox Rank Team Name norm. err pose norm. err rot Best Score 1 TangoUnchained 0.0179 0.0556 0.073498689 16 SCOUT Inc 0.0909 0.8357 0.926615725 35 baseline 0.3686 2.2038 2.572462691 Sunlamp Rank Team Name norm. err pose norm. err rot Best Score 1 lava1302 0.0113 0.0476 0.058860147 14 SCOUT Inc. 0.0832 1.0750 1.158212043 35 baseline 0.3736 2.2002 2.573856284 1 Burkhardt Z., Spessert, E., West, S., Gallucci, S., et al. “Trajectory Planning for a Proximity Operations Flyby Operation on the Tenzing Mission.” In AAS Guidance and Control Conference 2022. AAS-22-155. February 2022.
- 17. © 2023 SCOUT Inc Demo of SV-50 Inference 17
- 18. • Renders at ~3000 images/hour • Specific trajectory • Randomized ranges/pose/backgrounds • Earth or stars background for realistic image generation • Color/randomized image background for general training Creating a Model to Generate Synthetic Data: SCOUT’s System’s Capabilities 18 © 2023 SCOUT Inc
- 19. SCOUT Render Pipeline Demo 19 © 2023 SCOUT Inc
- 20. Demo of SV-50: Real-Time Inference 20 © 2023 SCOUT Inc
- 21. © 2023 SCOUT Inc Integrating Real Data and Conclusions 21
- 22. • Images are big vs. space downlink pipes • 9.6 kilobaud connections are very common • Emphasizing an iterative approach for data collection campaigns • Tenzig (2021): Noise + lens parameters • Near-term missions (2024): Target and SDA images in different lighting conditions Where’s the Real Data? 22 Right: Actual photo from SV-50 on Tenzing Left: Simulated image from SCOUT’s synthetics © 2023 SCOUT Inc
- 23. • Space is hard, not impossible • Synthetics are an inevitable part of space-based ML systems, so we have to learn to live with them • ML pipelines seem to generalize well from synthetics to physical data in the lab, given synthetic images with similar noise + aberrations • Standards and references for verification and testing are essential for deploying future machine vision systems in space (and on Earth!) Conclusions 23 © 2023 SCOUT Inc
- 24. • Flight experiments! 3 (!!!) SCOUT systems will fly in 2024 • Automated verification + validation pipelines • Learning pose estimation for arbitrary or damaged spacecraft (=unknown geometry a-priori) Future Work © 2023 SCOUT Inc 24
- 25. Synthetic Data: Tutorials and Examples 25 © 2023 SCOUT Inc Synthetic Data Resources SCOUT: Spacesight https://spacesight.scout.space/ Space ML https://spaceml.org/ Synthetic Data Tutorial https://bit.ly/synth-data SLAB Resources SLAB Website https://slab.stanford.edu/ SLAB Pose Estimation Paper arXiv:2203.04275v1 Robotic Testbed for Models arXiv:2108.05529v2
- 26. Media Slide – Thankyou/Contact Graphic 26 © 2023 SCOUT Inc
- 27. © 2023 SCOUT Inc Backup Material 27
- 28. SCOUT: Perception Systems 28 © 2023 SCOUT Inc
- 29. SCOUT vs. Traditional Approaches 29 Preprocessing Keypoint Extraction PnP Solver Pose Pipeline (ML) Images Pose Pose Images Traditional Pose Estimation © 2023 SCOUT Inc
- 30. 30 Previous Work: Proximity Guidance © 2023 SCOUT Inc
- 31. SCOUT: Remote-Sensing in Space On-Orbit Spacecraft Inspection 31 © 2023 SCOUT Inc
- 32. 1.Data continuity: the system must be able to handle drop-outs in detection from CV model 2.Data reliability: the system needs physically-informed models to mitigate false-positive or extremely inaccurate CV measurements 3.SCOUT has developed estimation filters which propagate target position/pose based on existing data and equations of motion across signal dropouts and which improve effective relative navigation accuracy Autonomous Edge System Considerations: Quality of Data 32 © 2023 SCOUT Inc
- 33. • Your system operates as expected in the simulated environment, how to improve confidence levels that system will operate as expected when deployed to real- world environment Evaluating Trustworthiness of Autonomous Machine Learning Systems 33 © 2023 SCOUT Inc
- 34. ESA Kelvin Pose Estimation Challenge: Loss Function/Scoring 34 © 2023 SCOUT Inc
- 35. • Lots of spacecraft, including Tango, exhibit various symmetries • "off by 90/180” errors are extremely easy to come by • Range is a major factor • <300 m: Fully resolved, maximum danger • >300 m: maybe partially resolved (can’t get orientation), less dangerous • >2 km: Non-resolved, dynamics less linear Loss Function Challenges 35 © 2023 SCOUT Inc
- 36. Determining Dataset Requirements: Resolution 36 © 2023 SCOUT Inc Lower resolution image of Tango spacecraft Higher resolution image of Tango spacecraft
- 37. Determining Dataset Requirements: Resolution vs. Exposure 37 © 2023 SCOUT Inc
- 38. Determining Dataset Requirements: Fidelity and Resolution – Exposure Time 38 © 2023 SCOUT Inc Underexposed image of Tango spacecraft Properly exposed image of Tango spacecraft exhibiting motion blurring