POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud Applications
Download as PPTX, PDF2 likes2,135 views
The document presents the challenges and methodologies for error detection and recovery in sporadic operations of cloud applications, focusing on the Process-Oriented Dependability (POD) approach. It outlines the significance of anomaly detection, undo/recovery planning, and the modeling/analysis of these operations to enhance cloud application reliability. Additionally, it discusses the role of process mining and AI planning in diagnosing errors during rolling upgrades in distributed systems.
POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud Applications
- 1. NICTA Copyright 2012 From imagination to impact POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud Applications Dr. Liming Zhu Liming.Zhu@nicta.com.au Principal Researcher, NICTA/UNSW April, 2014 at Berkeley AMPLab
- 2. NICTA Copyright 2012 From imagination to impact Outline • Dependable Cloud Operation • Approach: Process-Oriented Dependability (POD) – POD-Diagnosis – Undo/Recovery Planning using AI Planning – Modeling and Analysis using DTMC • Connections with AMPLab BDAS 2
- 3. NICTA Copyright 2012 From imagination to impact Dependable Cloud Operation: Motivation • Sporadic operations cause most outages – Deployment, reconfiguration, (rolling) upgrade, rollback… • as opposed to normal operations – DevOps-related: continuous integration/deploy/delivery • Etsy.com: 25 full deployments per day at 10 commits per deploy – Other drivers: resource sharing, micro services/partition migration, backup/recovery, auto-mitigation itself… • Limited control & visibility during sporadic operation – Heavy reliance on Cloud APIs – Limited visibility and exception handling capabilities 3
- 4. NICTA Copyright 2012 From imagination to impact Dependable Cloud Operation: Challenges • Our Context – Large-scale web/enterprise operation in Cloud – Distributed data analytics in Cloud (Hadoop/Spark) • Goal: detect, diagnose and react to errors occurring during a sporadic cloud operation • Challenges 1. Anomaly detection during sporadic operations 2. Undo/Recovery planning for recovery 3. Modelling and analysis of sporadic operation 4
- 5. NICTA Copyright 2012 From imagination to impact Sporadic Operation Example: Rolling Upgrade Update Auto-Scaling Group (ASG) Remove & Deregister Old Instances from ELB Wait for ASG to Start New Instances Terminate Old Instances Register New Instances with ELB Sort Instances Stop Start - Have 100 servers in cloud with version 1 software - Upgrade 10 servers at a time to version 2 software - No downtime or redundancy cost - Potentially take a long time to complete with errors during the operation with other interfering operations 5
- 6. NICTA Copyright 2012 From imagination to impact Challenge 1: Anomaly Detection • Traditional anomaly-based error detection is designed for “normal operation” – significant false positives OR disable all monitoring during sporadic operation • Continuous changes to the production systems – From months at scheduled downtime to hours at all times – Multiple operations at the same time • Quality of automation scripts + human – fully testing the operation (scripts + human) in uncertain cloud environment is very difficult 6
- 7. NICTA Copyright 2012 From imagination to impact Our Approach: Use Process Context • Offline: treat an operation as a process – Process discovered automatically from logs/scripts • Clustering of log lines and process mining – Intermediary step outcomes specified as assertions • Online: use process context – Process context: process/instance/step ids, expected states… – Errors detected by examining logs and monitoring data • Assertions evaluations integration with monitoring facilities • Compliance checking against expected processes using logs – Detected errors are further diagnosed for (root) causes • Examining a fault tree to locate potential root causes • Performing more diagnostic tests and on-demand assertions X. Xu, L Zhu, et. al. "POD-Diagnosis: Error Diagnosis of Sporadic Operations on Cloud Applications,” 44nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2014. 7
- 8. NICTA Copyright 2012 From imagination to impact Example: Rolling Upgrade Using Asgard Read by Operator Process Mining Service Controls Outputs Create SnapshotCheck AZs Create instance from snapshot Create AMI from instance Evaluate AMI Discovered Model Asgard Log dataLog dataGenerates Offline Online 8
- 9. NICTA Copyright 2012 From imagination to impact Process Mining Service: how it works • Process Mining: Discovery 1. Collect the logs (using Logstash) 2. Filter the logs 3. Calculating string distance (Levenshtein distance) between each pair of log lines 4. Cluster the log lines 5. Look at the dendrogram to decide on threshold 6. Name & combine clusters 7. Derive regular expressions for the clusters 8. Classify the log lines using the regular expressions and cluster names 9. Import altered log into process mining tools 10. Apply different process discovery algorithms 11. If anything requires changes, go back to the respective steps and redo from there 9
- 10. NICTA Copyright 2012 From imagination to impact POD-Detection: Error Detection Error Detection Service has two methods for detecting errors: • Assertion Checking • Conformance Checking 10
- 11. NICTA Copyright 2012 From imagination to impact Assertion Checking: how it works Log line: Assertions: 11
- 12. NICTA Copyright 2012 From imagination to impact Assertion Checking: how it works Log line: • Remove ... Assertions: • i has been de-registered from ELB • i has been removed from ASG • there is 1 less instance of v1 12
- 13. NICTA Copyright 2012 From imagination to impact Assertion Checking: how it works Log line: • Remove ... • Terminate ... Assertions: • i successfully terminated 13
- 14. NICTA Copyright 2012 From imagination to impact Assertion Checking: how it works Log line: • Remove ... • Terminate ... • Wait ... Assertions: • Next log line should appear within 17m35s (95 percentile) 14
- 15. NICTA Copyright 2012 From imagination to impact Assertion Checking: how it works Log line: • Remove ... • Terminate ... • Wait ... • New instance ... Assertions: • i„ successfully launched 15
- 16. NICTA Copyright 2012 From imagination to impact Conformance Checking: how it works Log lines: 16
- 17. NICTA Copyright 2012 From imagination to impact Conformance Checking: how it works Log lines: • Remove ... 17
- 18. NICTA Copyright 2012 From imagination to impact Conformance Checking: how it works Log lines: • Remove ... • Terminate ... 18
- 19. NICTA Copyright 2012 From imagination to impact Conformance Checking: how it works Log lines: • Remove ... • Terminate ... • Wait ... 19
- 20. NICTA Copyright 2012 From imagination to impact Conformance Checking: how it works Log lines: • Remove ... • Terminate ... • Wait ... • Terminate ...??? 20
- 21. NICTA Copyright 2012 From imagination to impact POD-Diagnosis: how it works • Fault trees are built as knowledge base • Process context used for fault tree pruning • On-demand diagnosis tests to locate the (root) causes 21
- 22. NICTA Copyright 2012 From imagination to impact Evaluation: POD-Detection/Diagnosis • Experiments – Rolling upgrade of 100+ node cluster in AWS • Fault injection+ confounding processes: random kill, scaling-in.. • Detected errors – Assertion checking: known errors and global errors • Examples: key management, launch configuration, images… – Compliance checking: unknown errors • skipping activities or undone activities • Time and precision – Compared with Asgard/Monitoring internal mechanisms • Detected more errors earlier – Diagnosis: limited to known causes in the fault tree • 95 percentile less than 4s; accuracy ranges 80%~100% 22
- 23. NICTA Copyright 2012 From imagination to impact Evaluation: POD-Detection/Diagnosis 23
- 24. NICTA Copyright 2012 From imagination to impact Other Related Research Challenges 1. Anomaly detection during sporadic operations 2. Undo/Recovery planning 3. Modelling and analysis of sporadic operation 24
- 25. NICTA Copyright 2012 From imagination to impact Challenge 2: Undo/Recovery Planning S1 S2 Serr A certain step Reparation Compensation Undo Parameterizable Redo Alternative Checkpoint-base Undo Previous states … ... S0S-i 25
- 26. NICTA Copyright 2012 From imagination to impact Undo/Undoability Approach in a Nutshell • Goal: undo support for “indirect control” setting – Problem 1: some actions are irreversible, e.g., delete – Problem 2: undo ≠ copy back previous state of memory • Have to call the right actions on the right resources in the right order – Problem 3: partly irreversible operations, e.g. on Amazon WS: • Stopping a machine disassociates an elastic IP address (if any), and releases internal IP / public DNS • Starting the machine isn‟t undo: elastic IP is dangling, internal IP / public DNS / timestamps are different • Solution components: Replace “do” with “pseudo-do” Undo System based on AI Planning • Outcome: sequence of undo actions Undoability Checking: • Is the operation I‟m about to execute undoable? • Learn which aspects can be fully undone for each operation (whole domain) • If not, can we abstract / change so that undoability is given? Projection (of a domain) 26 Ingo Weber et. al. Supporting undoability in systems operations. In USENIX LISA'13: Large Installation System Administration Conference, Washington, DC, USA, November 2013.
- 27. NICTA Copyright 2012 From imagination to impact Undoability Checking Approach Operation(s) to execute (e.g., script, command) Resources and properties required to be undoable Define Tool user (e.g., sys admin) Tool provider Full domain model (e.g., AWS) Projection Specification Generate Undoability CheckerDefine Apply Projection Generate Projected domain model Per operation: Generate pre and post-states Check undoability per pre-post state pair Undoability (yes/no) List of causes if not undoable Result Feedback For each pair: call AI Planner 27
- 28. NICTA Copyright 2012 From imagination to impact Challenge 3: Modeling and Analysis • Approach: Model as stochastic processes – Discrete/Continuous Markov Chain (DTMC/CTMC) • Forward states: net successful operations • Backward states: failure or deliberate rollback/undo • A family of g-k chains with different parameters – g: rolling-upgrade wave granularity. k: no. of failure/rollback per wave Daniel Sun & L Zhu, et. al. ” Understanding Rolling Upgrade” 33th International Symposium on Reliable Distributed Systems (SRDS), 2014 (submitted) 28
- 29. NICTA Copyright 2012 From imagination to impact Model used for Predictions - e.g. completion time, failure rate impact Optimization and Decision Problems - e.g. when to activate new versions to guarantee a 99.99% success 29
- 30. NICTA Copyright 2012 From imagination to impact Connection with AMPLab BDAS 30
- 31. NICTA Copyright 2012 From imagination to impact Projects Related to BDAS (1/2) 1. Log/Metrics analysis in POD-Diagnosis – Currently using Spark/MLBase – Voluminous log/events into Spark Streaming 2. Dependable deployment/operation of BDAS – POD applied to Hadoop before, maybe BDAS? 3. Multi-level granularity access for data analytics – Australian Urban Research Infrastructure Network (AURIN) • Portal to provide transport-related data to international researchers • Cluster sharing for in-portal pre-processing and analytics • de-anonymization concerns and different views for the same data – Evaluating how BDAS can support this 31
- 32. NICTA Copyright 2012 From imagination to impact Projects Related to BDAS (2/2) Redacted 4. Data scientist workflow and local exploration 5. Distributed machine learning 32
- 33. NICTA Copyright 2012 From imagination to impact Team Acknowledgement • Researchers – Len Bass – Alan Fekete – Anna Liu – Daniel Sun – Hiroshi Wada – Ingo Weber – Sherry Xu – Liming Zhu • Engineers – Adnene Guabtni – Chao Li • Students – Amer Abdalamer – Ahmed Alqahtani – Mostafa Farshchi – Min Fu – Jin Li – Matthew Sladescu – Donna Xu – DongYao Wu 33