Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
0 likes401 views
Autonomous systems present safety challenges due to their complexity and use of machine learning. Two key approaches are needed to address these challenges: (1) design-time assurance cases to validate safety requirements and (2) run-time monitoring architectures to detect unsafe behavior. Automated testing techniques leveraging metaheuristics and machine learning can help provide evidence for assurance cases and learn conditions to guide run-time monitoring. However, more industrial experience is still needed to properly validate these approaches at scale for autonomous systems.
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
- 1. Autonomous Systems: How to Address the Dilemma between Autonomy and Safety Lionel Briand ASE 2022 Keynote http://www.lbriand.info
- 3. Autonomous Systems âą Autonomy lies on a spectrum âą Fully autonomous systems Ăł advisory systems âą Open context: context not fully specified 3 Burton et al., 2019
- 4. Why Autonomy? âą Reactivity to events, e.g., lunar rover âą Alleviate cognitive load in complex tasks. E.g., adaptive cruise control âą New missions without human physical constraints, e.g., pilots in fighter jets âą Complete specifications difficult to formalize: environmental complexity as well as continual change 4
- 5. Autonomy and Machine Learning âą Machine learning (ML) helps with the implementation of the understanding and decision layers. âą ML: (1) infer input-output relationships from samples of intended behavior, (2) generalize the learned function to unknown and unforeseen inputs âą Risk: unanticipated responses and emergent system behavior âą Our focus: autonomous systems that use ML models 5
- 6. Semantic Gap âą Gap between intended functionality and specified functionality (Burton et al., 2019). âą More challenging for autonomous systems âą Causes: - Complexity and unpredictability of the operational domain - E.g., Urban environments 6
- 7. Semantic Gap âą Gap between intended functionality and specified functionality (Burton et al., 2019). âą More challenging for autonomous systems âą Causes: - Complexity and unpredictability of the system itself - E.g., Heterogeneous sensing channels 7
- 8. Semantic Gap âą Gap between intended functionality and specified functionality (Burton et al., 2019). âą More challenging for autonomous systems âą Causes: - Transfer of decision function to the system - E.g., ML functions do not deliver clear-cut answers 8
- 9. Paradox 10 âą âThe problem of deriving a suitable specification of the intended behavior is instead transferred to the problem of demonstrating that the implemented (learned) behaviour meets the intent.â Burton et al. 2019 âą Main challenge: Definition and validation of adequate system safety requirements
- 10. Assurance Cases âą A valid, evidence-based justification for a set of claims about the safety of a system for a given function over its operational context 11 Burton et al., 2019
- 11. Example: Autonomous Taxiing 12 Asaadi et al., 2020 CNN: âą Cross track error (CTE) âą Heading Error (HE)
- 12. Example: Autonomous Taxiing 13 âą Safety property: Lateral runway excursions âą Safety requirement: the aircraft CTE shall not exceed a specified offset whilst taxiing âą Hazard: aligning the aircraft nose with a different runway marking instead of the centerline âą Mitigation: retraining the ML component using training data augmented with appropriate test data exhibiting violations
- 13. Assurance Case 14 GSN notation: Goals, strategies, evidence Structured arguments backed up by evidence Evidence: âą artifacts âą verification Asaadi et al., 2020
- 14. Safety as a Control Problem 15 âą Safety should be treated as an emergent property of the system (Leveson, 2016) âą Paradigm Change:
- 15. Run-time Assurance Architecture 16 âą HiL simulator and iron bird âą CNN outputs used for run-time monitoring âą Contingency actions Asaadi et al., 2020
- 16. ML Challenges âą Robustness to adversarial or unexpected inputs âą Uncertainty of predictions âą Explainability: provides insights in a human interpretable way âą Verification: formal (proof) or experimental (test) 17 Tambon et al., 2022
- 17. Safety Verification âą Ideally: âą Formal safety guarantees âą Practical assumptions (about input space, model, properties âŠ) âą Scalable (e.g., to large models and systems) âą But âą We cannot have it all (e.g., high dimensionality, many parameters) âą Safety is not a model property, rather a system one 18
- 18. Safety Verification âą Formal analysis, e.g., reachability analysis âą Focus on robustness to perturbations âą Guarantees about models (not systems) under very restrictive assumptions (e.g., model architecture, input space shape, reasonable over-approximation of output space) âą Scalability issues âą Testing-based analysis, e.g., search-based testing âą Heuristics for input space exploration âą No guarantees, but no restrictive assumptions and more scalable âą Smart combination? 19
- 19. ML-Related Assurance Questions âą Model level: Under which input conditions is an ML- component classifying/predicting (in)correctly? âą System level: Under which conditions is an ML-component (un)safe for the system? âą Integration level: Are there possible undesirable interactions among (ML) components? âą Context: No specifications or code for ML components, increasingly large models (DL, RL) 20
- 20. Summary âą Autonomous safety functions can help achieve higher safety, e.g., emergency braking âą However, they also entail risks addressed by: (1) design-time assurance cases and (2) run-time safety monitoring play a key role to alleviate those risks âą We need automated support to collect appropriate evidence (1) and guide monitoring (2) are required in the context of ML- enabled autonomous systems. 21
- 21. Example Projects and Lessons Learned 22
- 23. Test Inputs: Adversarial or Natural? âą Adversarial inputs: Focus on robustness, e.g., noise or attacks âą Natural inputs: Focus on functional aspects, e.g., functional safety 24
- 24. Example: Key-points Detection âą DNNs used for key-points detection in images âą Many applications, e.g., face recognition âą Testing: Find test suite that causes DNN to poorly predict as many key- points as possible within time budget âą Images generated by a simulator 28 Ground truth Predicted
- 25. Example Application âą Drowsiness or gaze detection based on interior camera monitoring the driver âą In the drowsiness or gaze detection problem, each Key-Point (KP) may be highly important for safety âą Each KP leads to a test objective âą For our subject DNN, we have 27 test objectives âą Goal: Cause the DNN to mispredict as many key-points as possible âą Solution: Many-objective search algorithms (based on genetic algorithms) combined with simulator 30 Ul Haq et al., 2021
- 26. Overview 31 Input Generator (search) Simulator Input (vector) DNN Fitness Calculator Actual Key-points Positions Predicted Key-points Positions Fitness Score (Error Value) Most Critical Test Inputs Test Image
- 27. Results ⹠Our approach is effective in generating test suites that cause the DNN to severely mispredict more than 93% of all key-points on average ⹠Not all mispredictions can be considered failures ⊠(e.g., shadows) ⹠We must know when the DNN cannot be expected to be accurate and have contingency measures ⹠Some key-points are more severely predicted than others, detailed analysis revealed two reasons: ⹠Under-representation of some key-points (hidden) in the training data ⹠Large variation in the shape and size of the mouth across different 3D models (more training needed) 32
- 28. Interpretation âą Regression trees to predict accuracy based on simulation parameters âą Enable detailed analysis to find the root causes of high Normalized Error (NE) values, e.g., shadow on the location of KP26 is the cause of high NE values âą Regression trees show excellent accuracy, reasonable size. âą Amenable to risk analysis, gaining useful safety insights, and contingency plans at run-time 33 Image Characteristics Condition NE ! = 9 ⧠# < 18.41 0.04 ! = 9 ⧠# â„ 18.41 ⧠$ < â22.31 ⧠% < 17.06 0.26 ! = 9 ⧠# â„ 18.41 ⧠$ < â22.31 ⧠17.06 †% < 19 0.71 ! = 9 ⧠# â„ 18.41 ⧠$ < â22.31 ⧠% â„ 19 0.36 Representative rules derived from the decision tree for KP26 (M: Model-ID, P: Pitch, R: Roll, Y: Yaw, NE: Normalized Error) (A) A test image satisfying the first condition (B) A test image satisfying the third condition NE = 0.013 NE = 0.89
- 29. Real Images âą Test selection requires a different approach than with a simulator âą Labeling costs are significant 36
- 30. DNN Structural Coverage Criteria 37 Chen et al. 2020 âą Neuron coverage âą How neurons are activated âą Many variants
- 31. Limitations ⹠Require access to the DNN internals and sometimes the training set. Not realistic in many practical settings. ⹠There is a weak correlation between coverage and misclassification or poor predictions for natural inputs ⹠Also many questions regarding the studies focused on adversarial inputs ⊠38
- 32. Diversity-driven Test Selection âą Test inputs are real images âą Black-box approach based on measuring the diversity of test inputs. âą Scalable selection âą The more diverse, the more likely test inputs are to reveal faults 39 Agahababaeyan et al., 2022
- 33. Geometric Diversity (GD) âą Given a dataset X and its corresponding feature vectors V, the geometric diversity of a subset S â X is defined as the hyper-volume of the parallelepiped spanned by the rows of Vs, i.e., feature vectors of items in S, where the larger the volume, the more diverse is the feature space of S 40
- 34. Extracting Image Features âą VGG16 is a convolutional neural network trained on a subset of the ImageNet dataset, a collection of over 14 million images belonging to 22,000 categories. 41
- 35. Correlation with Faults Correlation between geometric diversity and faults
- 36. Pareto Front Optimization 44 âą Black-box test selection to detect as many diverse faults as possible for a given test budget âą Search-based Approach: Multi-Objective Genetic Search (NSGA-II) âą Two objectives (Max): âą diversity âą uncertainty (e.g., Gini) GD score Gini score
- 37. HUDD 45 âą How to help perform risk analysis with real images? âą Rely on heatmaps to generate clusters of images leading to failures because of a same root cause (common characteristics of the images) âą enable engineers to ensure safety with introducing countermeasures Error-inducing Test Set images Step1. Heatmap based clustering Root cause clusters C1 C2 C3 Step 2. Inspection of subset of cluster elements. Fahmy et al. 2021
- 38. âą Classification âą Gaze Detection 46 90 270 180 0 45 22.5 67.5 337.5 315 292.5 247.5 225 202.5 157.5 135 112.5 Top Center B o t t o m L e f t Bottom Center B o t t o m R i g h t Top Right Middle Right T o p L e f t Middle Left Example Application
- 39. Clusters identify different problems Cluster 1 (angle ~157.5) borderline cases Cluster3 (near closed eyes) incomplete training set 49 Cluster 2 (eye middle center) incomplete set of classes
- 40. SEDE: Simulator-based Explanations for DNN failurEs 50 âą Built on HUDD âą Generates readable descriptions of failure-inducing, real-world images âą Effective retraining âą Leverages availability of simulators Fahmy et al., 2022
- 41. SAFE: Black-Box Approach 51 Retraining Features Extraction Detection of root causes Error inducing images Data Preprocessing Features Extraction Clustering Root cause clusters Dimensionality Reduction Unsafe-set Selection Retraining Improved DNN âą Black-box alternative to HUDD âą No need to extend the DNN (LRP) âą Reduces training time and memory usage âą More accurate clustering (density based, DBSCAN) Attaoui et al., 2022
- 42. Reinforcement Learning Testing and Safety 52
- 43. Reward and Functional Faults 54 âą Testing goal: Detect and explain reward and functional faults âą A pole is attached to a cart, and the goal is to move the cart right and left to keep the pole from falling. âą Reward fault: the accumulated reward is less than a threshold, e.g., the pole falls down in the first 70 timesteps âą Functional fault: the pole is stable and, regardless of the accumulated reward, the cart moves beyond the 2.4 limited distance, and the episode terminates Functional fault Reward fault
- 44. STARLA: Objectives âą Detect as quickly as possible diverse faulty episodes âą Accurately characterize faulty episodes âą STARLA: Search and ML based 55 Zolfagharian et al. 2022
- 46. Testing via Physics-based Simulation 58 ADAS (SUT) Simulator (Matlab/Simulink) Model (Matlab/Simulink) âȘ Physical plant (vehicle / sensors / actuators) âȘ Other cars âȘ Pedestrians âȘ Environment (weather / roads / traïŹc signs) Test input Test output time-stamped output
- 47. Reinforcement Learning for ADAS Testing âą Use RL to change the environment in a realistic manner with the objective of triggering safety violations 59 Action (Behavior of environment actors, e.g., vehicle in front) Reward (e.g., based on distance from vehicle in front) State/Next State (Locations and conditions about the environment, e.g., collision) RL Agent RL-Environment Simulator (CARLA) Control (Image, location,âŠ) ADAS
- 48. Example Violation âą Violation: Ego Vehicle should not collide with other vehicle âą Vehicle-in-front slows down suddenly and then moves to the right âą Possible reason: Agent was not trained with such episodes 61 Car View Top View
- 49. System Testing Challenges 62 To address the challenges, we propose SAMOTA (Surrogate-Assisted Many-Objective Testing Approach) by leveraging many-objective search and Surrogate Models (SMs) Ul Haq et al. 2022 Large Input Space Many Safety Requirements Computationally- intensive Simulation
- 50. Surrogate Models âą Surrogate model: Model that mimics the simulator, to a certain extent, while being much less computationally expensive âą Research: Combine search with surrogate modeling to decrease the computational cost of testing (Ul Haq et al. 2022) 63 Polynomial Regression (PR) Radial Basis Function (RBF) Kringing (KR)
- 51. SAMOTA 64 Steps: 1. Initialization 2. Global search 3. Local search 4. Glocal search 5. Local search 6. ⊠Execute Simulator Global SMs Many Objective Search Algorithm Most Critical Test Cases Most Uncertain Test Cases Global Search Initialisation Execute Simulator Database Minimal Test Suite Local SM per Cluster Local Search Most Critical Test Cases Single Objective Search Algorithm Clustering Top Points
- 52. ML Impact on System Safety âą Inherent uncertainty in ML models âą Test mitigation mechanisms in systems to handle misclassifications or mispredictions âą Goal: We want to learn, as accurately as possible, when an ML component leads to system safety violations in terms of inputs and outputs âą Applications: This is expected to help guide and focus the testing of ML components or implement safety monitors for them. 65
- 53. Learn the Safety Envelope Monitoring: How far are we from the safety envelop for a given violation probability? 66 input output Safe Unsafe Learnt envelope
- 54. Conclusions 67
- 55. Safety Analysis âą The adoption of machine learning to enable autonomy raises new or amplify existing safety challenges âą Assurance cases (1) + run-time assurance architecture (2) âą Automated test support to collect evidence (1) and learn safety conditions to monitor (2) âą Approaches based on metaheuristic search and machine learning have shown to be practical, effective, and reasonably efficient âą Limited industrial experience properly reported on these topics 68
- 56. Automated Testing ⹠Remains key mechanism to provide safety evidence ⹠Different levels of testing: model, integration, system ⹠Automation is key at all levels and requires different strategies ⹠Strategy: evolutionary computing and machine learning ⹠Scalability is the main challenge: simulations, surrogate models, ⊠⹠More work needed beyond stateless models (DNN): reinforcement learning ⊠69
- 57. Autonomous Systems: How to Address the Dilemma between Autonomy and Safety Lionel Briand ASE 2022 Keynote http://www.lbriand.info
- 58. References 72
- 59. Selected References âą Ul Haq et al. "Automatic Test Suite Generation for Key-points Detection DNNs Using Many-Objective Search" ACM International Symposium on Software Testing (ISSTA), 2021 âą Ul Haq et al., âEfficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimizationâ IEEE/ACM ICSE 2022 âą Fahmy et al. "Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning" IEEE Transactions on Reliability, Special section on Quality Assurance of Machine Learning Systems, 2021 âą Fahmy et al. "Simulator-based explanation and debugging of hazard-triggering events in DNN-based safety-critical systemsâ, ArXiv report, https://arxiv.org/pdf/2204.00480.pdf, ACM TOSEM, 2022 âą Attaoui et al., âBlack-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clusteringâ, ArXiv report, https://arxiv.org/pdf/2201.05077v1.pdf, ACM TOSEM, 2022 âą Aghababaeyan et al., âBlack-Box Testing of Deep Neural Networks through Test Case Diversityâ, https://arxiv.org/pdf/2112.12591.pdf âą Zolfagharian et al., âSearch-Based Testing Approach for Deep Reinforcement Learning Agentsâ, https://arxiv.org/pdf/2206.07813.pdf 73
- 60. Selected Safety References âą Asaadi et al., âAssured Integration of Machine Learning- basedAutonomy on Aviation Platformsâ, 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC) âą Burton et al., âMind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspectiveâ, Artificial Intelligence (Elsevier), 2019 âą Tambon et al., âHow to certify machine learning based safety-critical systems? A systematic literature reviewâ, Automated Software Engineering (Springer), 2022 âą Leveson, âEngineering a safer world: Systems thinking applied to safetyâ, The MIT Press, 2016. 74
- 61. Selected ML Testing References âą Goodfellow et al. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014). âą Zhang et al. "DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems." In 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2018. âą Tian et al. "DeepTest: Automated testing of deep-neural-network-driven autonomous cars." In Proceedings of the 40th international conference on software engineering, 2018. âą Li et al. âStructural Coverage Criteria for Neural Networks Could Be Misleadingâ, IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (NIER) âą Kim et al. "Guiding deep learning system testing using surprise adequacy." In IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019. âą Ma et al. "DeepMutation: Mutation testing of deep learning systems." In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), 2018. âą Zhang et al. "Machine learning testing: Survey, landscapes and horizons." IEEE Transactions on Software Engineering (2020). âą Riccio et al. "Testing machine learning based systems: a systematic mapping." Empirical Software Engineering 25, no. 6 (2020) âą Gerasimou et al., âImportance-Driven Deep Learning System Testingâ, IEEE/ACM 42nd International Conference on Software Engineering, 2020 75