Machine Learning for Continuous Delivery
- 1. Continuous Delivery & Machine Learning – SEA Edition Ravi Lachhman Harness.io @ravilach
- 2. Who am I? <3 SRE
- 4. 5 Dozens of Flavors People in need of SUGAR
- 5. Eaten No Negative Yelp Reviews More are bought What is Winning Doughnuts?
- 6. Doughnut Deployment Pipeline Idea You VerifyTestBuild Manual RollbackDeploy Yum New Treat Hours MinsDays Mins Ask co-workers about taste Blast on Social Media Gather Raw Material Prepare New Doughnut Flavor Eat / Instagram Doughnut Yelp / Google Map Krispy Kreme Subjective Look Here Ouch
- 7. Wait we are in Software…
- 10. Copyright © Harness 2018 Confidential 11 Deploying & Verifying Apps Is A Little Different Press Monthly Deployment Console
- 11. Confidential Traditional Verification “Application is up so everything is fine” “Homepage Works” “Not seeing an alerts” “No-one has complained yet”
- 12. Continuous Delivery = Automate Everything Dev Stage 1 Static Code Analysis Unit Tests Integration Tests Depend/license Tests Deployment Pipeline CustomerCode QA Stage 2 Smoke Tests Regression Tests Security Tests Browser Tests Load/Perf Tests On Success Production Stage 3 Availability Tests Performance Tests Quality Tests Security Tests Canary Verification On Success On Commit
- 14. Disclaimer: Machine Learning Is Not A Silver Bullet (But its better at doing some things better than humans)
- 15. Unsupervised Machine Learning Algorithms identify meaning and context of data on its own E.g. Natural Language Processing (NLP), Neural Nets Supervised Machine Learning Algorithms require human assistance Neural Feedback (Ignore, Dismiss, Ratings, Yes, No) NOT….Rules Regex, Thresholds, Configuration or Standard Deviation! Machine Learning 101
- 16. Applying Machine Learning Performance Quality Security Availability Deploy Rollback Continuous Verification Performance Quality Business Availability Unsupervised Machine Learning Automate Deployment Test/Verify/Rollback $ Revenue / Throughput
- 17. Context Application Artifact (Service) Infrastructure Deployment Context, Control and Test Data Control Data (Baseline) Previous Tests Previous Deployments Previous Canary Phases Test Data (New) New Tests New Deployments New Canary Phases
- 18. Verify your Metrics + Custom Metrics
- 22. Metric = Timestamp + Name + Value E.g. “1/1/19 10:01:34, CPU, 99.9%” This is what Dynatrace, AppDynamics, New Relic, Datadog and Prometheus Provides Analyze Time-Series Metrics from APM/Monitoring
- 23. Symbolic aggregate representation (SAX) Machine Learning Algorithms Used (time-series) https://jmotif.github.io/sax-vsm_site/morea/algorithm/SAX.html
- 24. Machine Learning Algorithms Used (time-series) Hidden Markov Models https://en.wikipedia.org/wiki/Hidden_Markov_model Predict when data is incomplete
- 25. Verify your Logs
- 26. Confidential 27 Stack Trace Error Regressions Quality Regression
- 27. Copyright © Harness 2018 Confidential 28 KMeans Clustering Group Events By Natural Language Similarity
- 28. Automate Verification (Unstructured Events) Usual crap AnomaliesUsual crapAnomalies
- 29. Entropy (Frequency of Attributes) 0 1 High EntropyLow Entropy
- 30. Application Log File has 100 Events 10 of those are DoughnutNotFound Exceptions (Cluster) DNF are thrown in 10 different places of the application (different positions relative to each other) Com.app.package.classA Com.app.package.classB Com.app.package.classC Simple Example
- 31. Jaccard & Cosine Distance
- 33. KMeans Clustering isn’t Perfect Event Data can contain noise (guids, urls, ip addresses, cookies, …)
- 34. Use Trained Neural Nets to Reduce False Positives Non-Neural Net This is why Engineers Don’t Trust Machine Learning Trained Neural Net
- 35. Rollback in 32 Seconds
- 36. Build confidence, let machines help Fold early, fold often Doughnuts are amazing What did we learn?
- 37. Copyright © Harness 2019 Confidential 38 Questions? Free 15-Day Trial at www.harness.io
- 38. Grab the slides! Free 15-Day Trial at www.harness.io Ravi.Lachhman@harness.io