Scenario-Based Development & Testing for Autonomous Driving

Scenario-Based Development &
Testing for Autonomous Driving
Yu Huang
Sunnyvale, California
yu.huang07@gmail.com

Outline
´ Formal Scenario-Based Testing of Autonomous Vehicles: From
Simulation to the Real World, 2020
´ A Scenario-Based Development Framework for Autonomous Driving,
2020
´ A Customizable Dynamic Scenario Modeling and Data Generation
Platform for Autonomous Driving, 2020
´ Large Scale Autonomous Driving Scenarios Clustering with Self-
supervised Feature Extraction, 2021
´ Generating and Characterizing Scenarios for Safety Testing of
Autonomous Vehicles, 2021
´ Systems Approach to Creating Test Scenarios for Automated Driving
Systems, Reliability Engineering and System Safety (215), 2021

Formal Scenario-Based Testing of Autonomous
Vehicles: From Simulation to the Real World
´ It is the approach to automated scenario-based testing of the safety of autonomous
vehicles, especially those using advanced artificial intelligence-based components,
spanning both simulation-based evaluation as well as testing in the real world.
´ This is based on formal methods, combining formal specification of scenarios and
safety properties, algorithmic test case generation using formal simulation, test case
selection for track testing, executing test cases on the track, and analyzing the
resulting data.
´ SCENIC is a domain-specific probabilistic programming language for modeling the
environments of cyber-physical systems.
´ A SCENIC program defines a distribution over scenes, configurations of objects and
agents; a program describing “bumper-to-bumper traffic” might specify a particular
distribution for the distance between cars.
´ SCENIC provides convenient syntax for geometry, along with declarative constraints,
which together make it possible to define such complex scenarios in a concise,
readable way.

´ One is to construct tests from scenarios created from crash data analysis and
naturalistic driving data (NDD) analysis, which leverage human driving data to
generate test scenarios.
´ The PEGASUS project focuses on (i) bench-marking human driving performance using
a dataset comprising crash reports, NDD, etc., (ii) characterizing requirements that AVs
should satisfy to ensure the traffic quality is at least unaffected by their presence.
´ OpenSCENARIO defines a file format for the description of the dynamic content of
driving and traffic simulators, based on the extensible markup language (XML).
´ GeoScenario is a somewhat higher-level domain specific language (DSL) for scenario
representation, whose syntax also looks like XML.
´ SCENIC is a flexible high-level language that is complementary to these,
complemented by the open-source VERIFAI toolkit.
´ The Measurable Scenario Description Language (M-SDL) is a recent higher-level DSL
similar to SCENIC, which precedes its definition; while M-SDL is more specialized for AV
testing, it has less support than SCENIC for probabilistic and geometric modeling and is
not supported by open-source back-end tools for verification, debugging, and
synthesis of autonomous AI/ML based systems.

´ The VERIFAI toolkit provides a unified framework for the design and analysis of AI- and
ML-based cyber-physical systems, based on a simple paradigm: simulations driven by
formal models and specifications.
´ In VERIFAI, first parametrize the space of environments and system configurations of
interest, either by explicitly defining parameter ranges or using the SCENIC language
described above.
´ VERIFAI then generates concrete tests by searching this space, using a variety of
algorithms ranging from simple random sampling to global optimization techniques.
´ Each test results in a simulation run, where the satisfaction or violation of a system-level
specification is checked; the results of each test are used to guide further search, and
any violations are recorded in a table for further analysis.
´ This architecture enables a wide range of use cases, including falsification, fuzz
testing, debugging, data augmentation, and parameter synthesis.

´ The LGSVL Simulator is an open-source autonomous driving simulator used to
facilitate the development and testing of autonomous driving software systems.
´ With support for the Robot Operating System (ROS, ROS2), and alternatives such as
CyberRT, the simulator can be used with popular open source autonomous platforms
like Apollo (from Baidu) and Autoware (from the Autoware Foundation).
´ Thus, it allows one to simulate an entire autonomous vehicle with full sensor suite in a
safe, deterministic (assuming determinism in AV software stack behavior), and
realistic 3D environment.
´ The LGSVL Simulator provides simultaneous real-time outputs from multiple sensors
including cameras, GPU-accelerated LiDAR, RADAR, GPS, and IMU.
´ Environmental parameters be changed including map, weather, time of day, traffic
and pedestrians, and the entire simulation can be controlled through a Python API.
´ The LGSVL Simulator is open-source on GitHub (https://github.com/lgsvl/simulator).

´ GoMentum Station is the largest secure AV test site in the United States.
´ Located in Concord, CA, 35 miles from San Francisco and 60 miles from Silicon Valley,
GoMentum features 19 miles of roadways, 48 intersections, and 8 distinct testing
zones over 2,100 acres, with a variety of natural and constructed features.
´ Vehicle manufacturers, AV system developers and other entities have been testing
connected and automated vehicles at GoMentum since 2014.
´ Experiments were conducted in the “urban” or “downtown” zone of GoMentum,
which features mostly flat surface streets that are approximately 20 feet wide and
several signed, unsigned, and signalized intersections amid an urban and suburban
landscape with buildings, trees, and other natural and human-made features.
´ Speeds are restricted to under 30 mph. The roads feature clear and visible lane
markings on freshly paved roads.
´ The traffic signs include one-way, stop, yield, and speed limits.

Formal scenario-based testing methodology
Create Simulation Model Formalize Test Scenario Formalize Safety Property/Metric
Identify Safe/Unsafe Test Cases Select Test Cases for Track Testing Implement Selected Test Cases on Track
Record Results and Perform Data Analysis

A Scenario-Based Development
Framework for Autonomous Driving
´ “Scenarios” were used to describe the use of the system, the requirements for use, the use
environment, and the construction of more feasible systems.
´ The autonomous driving scenario can be understood as such: a scenario is the dynamic
description of the components of the autonomous vehicle and its driving environment
over a period of time.
´ The scene can be regarded as a combination of the driving situation and driving scene of
an autonomous vehicle.
´ This article summarizes the research progress of scenario-based testing and development
technology for autonomous vehicles.
´ It systematically analyzed previous research works and proposed the definition of
scenario, the elements of the scenario ontology, the data source of the scenario, the
processing method of the scenario data, and scenario-based V-Model.
´ It summarized the automated test scenario construction method by random scenario
generation and dangerous scenario generation.

Ontology for Scenario Elements
Scenario Data Source
• Determining the ontology of the scenario
element is the cornerstone of scenario-
based techniques.
• Commonly used open source schemas
such as OpenDrive and OpenScenario
specified their road elements and traffic
dynamic elements definitions in detail.
• The test vehicle will have a significant
impact on surrounding scene elements,
especially other traffic participants.
• The interaction between the test vehicle
and the surrounding driving environment
forms a closed loop.

Scenario Data Processing Flow
Scenario-based V-Model. The scenario database is embedded in
all stages of the development, where the scenario extraction is
mostly achieved by search interfaces.
The V-model includes virtual testing, such as software-in-
the-loop testing (SIL), hardware-in-the-loop testing (HIL),
and real road testing, such as close field testing and open
road testing.

´ The generation methods mostly fall into two categories: random scenario generation
and dangerous scenario generation.
´ The random scenario generation methods mainly lies in three categories. 1) Random
sampling represented by Monte Carlo sampling and fast random search tree. 2)
Importance based sampling such as importance level analysis of scene elements. 3)
Machine learning based methods.
´ First of all, it is necessary to define and classify dangerous scenes. Many projects have
conducted research on car dangerous scenes.
´ There are three technical challenges for auto test scenario generation: authenticity,
granularity, and measurement.
´ In order to ensure the authenticity of the scene during the virtual test, the reference
measurement system (RMS) should be established during the virtual scene test.
´ The granularity of scene elements needs to be adapted according to technological
development.
´ Collision is often used as the measurement for the virtual test.

A Customizable Dynamic Scenario Modeling and
Data Generation Platform for Autonomous Driving
´ Safely interacting with humans is a significant challenge for autonomous driving.
´ The performance of this interaction depends on machine learning-based modules of an
autopilot, such as perception, behavior prediction, and planning.
´ These modules require training datasets with high-quality labels and a diverse range of realistic
dynamic behaviors.
´ Consequently, training such modules to handle rare scenarios is difficult because they are, by
definition, rarely represented in real-world datasets.
´ Hence, there is a practical need to augment datasets with synthetic data covering these rare
scenarios.
´ In this paper, present a platform to model dynamic and interactive scenarios, generate the
scenarios in simulation with different modalities of labeled sensor data, and collect this
information for data augmentation.
´ This the first integrated platform for these tasks specialized to the autonomous driving domain.

Overview of the platform’s pipeline

Part of a SCENIC program modeling a badly-parked
car pulling into the ego’s driving lane
SCENIC is a formal scenario description
language developed to model static and
dynamic, multi-agent scenarios to help
designing and testing autonomous systems
such as self-driving cars. SCENIC is a
probabilistic programming language that
enables users to specify distributions over
scenes (i.e., configurations of objects in
space as well as their features) and dynamic
behaviors. Furthermore, users can construct
objects in an intuitive imperative style while
simultaneously imposing declarative hard
(i.e., strict) and soft (i.e., probabilistic)
constraints. SCENIC’s concise, readable
syntax simplifies modelling spatial and
temporal relationships.

Collected Sensor Data and Labels using the SCENIC program (see the last page)

Large Scale Autonomous Driving Scenarios
Clustering with Self-supervised Feature Extraction
´ The clustering of autonomous driving scenario data can substantially benefit the
autonomous driving validation and simulation systems by improving the simulation tests’
completeness and fidelity.
´ This article proposes a comprehensive data clustering framework for a large set of vehicle
driving data.
´ This approach thoroughly considers the traffic elements, including both in-traffic agent
objects and map information.
´ Meanwhile, proposed a self-supervised deep learning approach for spatial and temporal
feature extraction to avoid biased data representation.
´ With the newly designed driving data clustering evaluation metrics based on data-
augmentation, the accuracy assessment does not require a human-labeled dataset,
which is subject to human bias.
´ Via such unprejudiced evaluation metrics, the approach surpasses the existing methods
that rely on handcrafted feature extractions.

The pipeline of vehicle driving data clustering. The information of both vehicle trajectories and map is
first combined and unified through renderings. The rendering of each frame is first compressed for
compact representation through CNN-VAE model as frame feature vector. A sequence compression
with RNN model further compresses a sequence of frame feature vectors into a sequence feature
vector. A clustering algorithm then groups such compact sequence feature representations. The
clustered sequences are visualized by simulation platform of Apollo.

Here shows the image representation for two consecutive frames. According to the Frenet coordinate, the
position of the ego vehicle in the first layers remain the same, while only the color value of the pixels vary
frame by frame, indicating the change of ego vehicle speed. The numbers and locations of the agent vehicles
differ frame by frame, as shown in layers 2 and 3. Layer 4 displays the lane boundaries information in the
surrounding environment. The orientation of the lanes rotates to cope with the Frenet coordinate system.

(a) a common process of preparing input and
supervision data pairs for training prediction
model; (b) the way to construct input and
supervision data pairs for model training.
Augmentation of a data sequence into
multiple same cluster data sequence

Generating and Characterizing Scenarios
for Safety Testing of Autonomous Vehicles
´ Extracting interesting scenarios from real world data as well as generating failure cases is
important for the development and testing of autonomous systems.
´ It proposes efficient mechanisms to both characterize and generate testing scenarios
using a state-of-the-art driving simulator.
´ For any scenario, this method generates a set of possible driving paths and identifies all
the possible safe driving trajectories that can be taken starting at different times, to
compute metrics that quantify the complexity of the scenario.
´ It tests to characterize real driving data from the Next Generation Simulation (NGSIM)
project, as well as adversarial scenarios generated in simulation.
´ It ranks the scenarios by defining metrics based on complexity of avoiding accidents and
provide insights into how the AV could have minimized the probability of incurring an
accident.
´ It demonstrates a correlation between the metrics and human intuition.

quantized region on the map where the AV can
travel in the next time step based on maximum
allowed steering and acceleration
tensor representation of AV state at time ti
during the scenario roll-out
pre-calculation of map matrix and
collision tensor from a scenario
Scenario scoring based on
search for safe driving policy

´ A driving scenario includes the description of the environment, the initial states of all
actors at time t0, and the driving policies of all actors except for the AV.
´ Such a definition allows using a scenario to test or benchmark different AV policies.
´ They use the term sequence to indicate a temporal succession of states of all the
vehicles.
´ A sequence can be obtained by executing a scenario on a simulator using a
specific AV policy or obtained directly from real data.
´ They define the metrics below, with the first two metrics computed directly from the
number of safe paths and the number of on-road paths, and the rest computed
using a similar propagation in tensor space:
´ SafePathInv (1/=#p)
´ UnsafePercent (pc%)
´ AvgEffort (E[es])
MinEffort (min[es])
NarrowInv (1/E[min[cs]]):

´ A base sequence is simulated by spawning an AV and other vehicles on the map at
random initial states (positions and velocities), and assigning to each vehicle
(including the AV) a simple driving policy described by a Markov chain where
vehicles change lane or speed with probability p after every N time steps, and
otherwise drive on the same lane.
´ To generate an unsafe sequence from the base sequence, change the policy of a
vehicle that is closest to the AV.
´ The selected vehicle sets its steering and acceleration to decrease the distance to
the AV and increase the chance of a collision (e.g. simulating a distracted driver).
´ The driving policy for this vehicle during this time is determined by computing
acceleration and steering such that the distance to the AV decreases.
´ To ensure that the generated adversarial scenarios are more realistic, seed this
generator from real driving scenarios.
´ Starting from a real scenario, modify the behavior of one driver for 3 seconds to
generate an incident.

Score and metric break down of (a) scenarios from NGSIM,
and (b) generated adversarial scenarios with accidents

Systems Approach to Creating Test
Scenarios for Automated Driving Systems
´ Increased safety has been advocated as one of the major benefits of the introduction of
Automated Driving Systems (ADSs).
´ To prove ADSs are safer than human drivers, some work has suggested that they will need
to be driven for over 11 billion miles.
´ Types of scenarios encountered by ADSs during testing are critically important.
´ With a Hazard Based Testing approach, this paper proposes that the extent to which
testing miles are ‘smart miles’ that reflect hazard-based scenarios relevant to the way in
which an ADS fails or handles hazards is a fundamental, if not pivotal, consideration for
safety-assurance of ADSs.
´ Using Systems Theoretic Process Analysis (STPA) method as a foundation, an extension to
the STPA method has been developed to identify test scenarios.
´ The approach has been applied to a real-world case study of a SAE Level 4 Low-Speed
Automated Driving system (a.k.a. a shuttle).

´ In order to ensure that the systems have a safe and a robust functionality, it is
important to be able to define test scenarios which are able to: 1) trigger real-world
use sequence 2) represent user input values 3) define and identify “all” operating
conditions.
´ For ADASs and ADSs, it is more important to identify “how a systems fails (or
misbehaves)” as compared to “how a system works”.
´ In order to identify the smart miles, a Hazard Based Testing (HBT) approach was
introduced to complement the traditional requirements based testing approach
used in the automotive domain.
´ Three steps of HBT: 1. Identification of hazards 2. Creating test scenarios for the
identified hazards 3. Pass criteria for the created test scenarios
´ There are various hazard analysis methods, i.e. Failure modes and effects analysis
(FMEA), Fault Tree Analysis (FTA), Event Tree Analysis (ETA), Hazard and operability
studies (HAZOP) and Systems Theoretic Process Analysis (STPA).

´ STPA is a hazardous event identification method which is based on the accident
causality model called STAMP (Systems Theoretic Accident Model and Processes)
which in turn is grounded in systems theory and control theory.
´ Unlike other methods like FMEA, FTA and ETA which are focused on identifying
downstream effects of failures or chain of events, STPA addresses failures as well as
non-failures that lead to losses including interactions among components and
systems operating exactly as designed and providing functions exactly as specified.
´ One of the significant benefits of using STPA is its capacity to identify diverse causal
factors (component failures, component interactions, requirements flaws, human-
errors, design flaws, societal issues, organizational issues.
´ STPA is a four step process. The 1st step is to define the purpose of the analysis. The
2nd step is to build a model of the system called a control structure. The 3rd step
identifies Unsafe Control Actions (UCAs). The 4th step identifies loss scenarios.

´ A complete test scenario description will have four components: 1. Scenery 2.
Environment 3. Dynamic elements 4. Pass criteria 5. Additional context.
´ Scenery defines all geo-spatially stationary objects in the Operational Design Domain
(ODD) of the vehicle, and includes attributes like road layouts, road furniture (e.g.
barricades), traffic lights etc., thus scenery can be sub-categorized by various
attributes (with their sub-attributes).
´ Environment includes attributes such as weather, visibility, connectivity etc. Selection
of the scenery and environment conditions to be used for testing is influenced by the
ODD of the System-Under-Test (SUT).
´ All moving objects and actors in the world comprise the dynamic elements category.
´ The pass criterion defines the set of conditions (internal to SUT or external to SUT) for
which the test scenario will be considered as a pass.
´ Additional context refers to the context element of the UCA and the causal factors
for “Potential control action not followed/How could this happen”.

Test scenario attributes for parameterization (top level)

Dynamic element attributes for parameterization (top level)

LSAD: Low-Speed Automated Driving

Scenario-Based Development & Testing for Autonomous Driving

Scenario-Based Development & Testing for Autonomous Driving

More Related Content

What's hot (20)

Similar to Scenario-Based Development & Testing for Autonomous Driving (20)

More from Yu Huang (20)

Recently uploaded (20)

Scenario-Based Development & Testing for Autonomous Driving