Improving Adobe Experience Cloud Services Dependability with Machine Learning
2 likes1,494 views
The document discusses improving the reliability of Adobe Experience Cloud services through machine learning, detailing an integrated approach to marketing technology that enables real-time decision-making and capacity planning. It highlights the complexity introduced by microservices, the need for better risk classification of changes, and the development of models to reduce incidents during releases. Future work includes linking monitoring systems, ticketing systems, and CI/CD workflows for enhanced context and service dependency awareness.
Improving Adobe Experience Cloud Services Dependability with Machine Learning
- 1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Improving Adobe Experience Cloud Services Dependability with Machine Learning Nicolas Brousse | Director, SRE
- 2. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Cloud — an all-in-one experience solution Adobe Experience Cloud brings together all of your marketing tech in a single place, so you can do everything from managing your content and delivering email campaigns to automating your ad buying and measuring your success. One integrated approach for one seamless experience. 2
- 3. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3 Dependable Services
- 4. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4 Real Time Decisions
- 5. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5 Increase In Complexity
- 6. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6 Growing Amount Of Data
- 7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 7 Core Infrastructure ML & Data Pipeline Adobe Advertising Cloud Enable a Machine Learning Pipeline supported by a Kubernetes Infrastructure to allow fast iterations and deployment of new models to production.
- 8. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8 Adobe Analytics Build models to provide traffic prediction and improve capacity planning efforts. Forecasting & Capacity Planning
- 9. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9 Adobe Experience Platform Provide anomaly detection on streamed live timeseries data to enable auto- remediation rules based of forecasted dynamic thresholds. Anomaly Detection & Self-Healing
- 10. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 10 Risk Classification & Failure Prediction Adobe Experience Cloud Improve change management processes through dynamic risk classification of changes to reduce number of incidents due to changes.
- 11. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 11 Build & Deploy Measure & Plan Verify & Test Operate & Monitor
- 12. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Replacing Human Intuition with Machine Learning Toward A Better Notion Of Risk During A Release
- 13. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adoption of Microservices and Distributed Systems means: § Increased complexity for Developers § Increased complexity for Operators § Increased of noise in DevOps communications 13 Complexity Microservices & Distributed Systems Human Cognitive Limit
- 14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Analyze Changes and Likelihood of Leading to An Incident § Goal: Reduce Incidents due to Changes With Better Risk Classification of Changes § Exported Data of 123 Incidents due to Change and 40,991 Normal Changes from January 2017 to June 2018 § Use of Jupyter Notebooks with Python and scikit-learn, etc. 14
- 15. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Data Findings § Most Changes Leading to an Incident are self reported as Low Risk § Most Changes Leading to an Incident have a short approval deadline <2H § Incident Occurrence Increases Proportionally to Maintenance Duration 15
- 16. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Models Comparison 16 Overall Accuracy Incident Accuracy Notification Noise Logistic Regression 85.79% 76.92% 28% ANN using MLP 83.09% 94.06% 13.8% Linear SVM 96.04% 99.26% 4.8%
- 17. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Future Work § Link to Existing Monitoring Systems for real-time feedback loop § Link to Tickets System and Git repository for better context analysis § Link to Service Discovery for Service Dependency awareness § Add checkpoints into CI/CD workflow 17