Measuring Software Performance of Self-Driving Vehicles with Scenario-Based Simulation
0 likes13 views
The document presents a framework for measuring the performance of self-driving vehicles through scenario-based simulation, emphasizing the need for multi-modal approaches tailored to specific operational domains. It discusses the importance of realistic simulation fidelity, safety criteria, and the collaboration between industry, regulatory, and academic partners for standardization. Challenges include defining coverage, ensuring consistency in test results, and aligning on safety metrics as the technology evolves.
Measuring Software Performance of Self-Driving Vehicles with Scenario-Based Simulation
- 1. Measuring software performance of self-driving vehicles with scenario-based simulation Shalin Mantri, Product Lead - Uber ATG October 23, 2018 ● AV Test & Development Symposium
- 2. 01 Motivation 02 Objectives 03 Challenges Agenda
- 4. To demonstrate the expected performance of an ADS for deployment on public roads, test approaches may include a combination of simulation, test track, and on-road testing. U.S. DOT
- 5. Testable, repeatable, flexible. Achieves coverage on the ODD. Measure performance. Multi modal, not one size fits all. Shared framework across the industry. Faithfully models the real world. Simulation Objectives
- 6. Leverage an appropriately scoped modality for the component(s) of the system that are being validated. Beware of “one size fits all” solutions. Multi modal, not one size fits all 1 Perception Prediction Motion Planning Controls
- 7. Hardware Modified base Vehicle Platform with Uber-specific mounting provisions, electrical harness, cooling interface, interior trim, and software control API Gateway module serves as a bridge between the base Vehicle Platform and the Uber Self Driving System, translating messages and commanding the vehicle’s actuators (brakes, throttle, steering) Sensors provide a 360° 3-dimensional view of the environment surrounding the vehicle Custom designed compute and storage allow for real-time processing of data while a fully integrated cooling solution keeps components running optimally Collision Mitigation (CM) system operates independently as a safeguard for certain situations requiring activation of the vehicle braking system
- 8. Software AVMaps and Localization Framework Computing Sensors Perception Prediction Motion Planning Controls Vehicle
- 9. Log Sim Logged Sensors Perception Prediction Motion Planning Controls Vehicle Pose Vehicle Model
- 10. Perception output Sim Engine Prediction Motion Planning Controls Virtual Sim Vehicle Pose Vehicle Model
- 13. Vehicle Miles Traveled Disengagements Infractions Driving Behavior Rider Comfort Crashes Many Road Driving Measures in the Industry
- 14. Vehicle Miles Traveled Disengagements Infractions Driving Behavior Rider Comfort Crashes … And Simulation is a Method for Measuring Performance
- 15. Fidelity is important when the stakes are large (high-frequency scenarios or rare, harmful events). Equally important to know when the simulation isn’t trustworthy. Faithfully models the real world 3
- 17. It’s not about millions of miles driven in simulation; it’s about the right miles and the right scenarios that we intend to handle in an operational domain. Achieves coverage on the ODD 4
- 19. Each sim has safety-derived pass/fail criteria that can be computed reliably with consistent results across executions. This criteria will evolve as we learn from past experience. Testable, repeatable, flexible 5
- 20. Safety is not a competitive advantage. We want to work closely with regulatory, academic, and industry partners. Shared frameworks 6
- 21. Testable, repeatable, flexible. Achieves coverage on the ODD. Measure performance. Multi modal, not one size fits all. Shared framework across the industry. Faithfully models the real world. Simulation Objectives
- 22. Testable, repeatable, flexible. Moving from custom-defined safety buffers to algorithmically computed ones. Frequency of software changes. Achieves coverage on the ODD. Need formal taxonomy of ODD. Many definitions of “coverage”. Expensive to achieve true scale. Measure performance. Need alignment on safety-relevant measures, including leading indicators. Need alignment on the role of simulation. Multi modal, not one size fits all. Evolving system architectures. Bespoke and proprietary sim modalities. Shared framework across the industry. Still developing across a number of areas... Faithfully models the real world. Measuring accuracy is a moving target. What is “good enough”? Simulation Objectives… and Challenges