Machine Learning to Turbo-Charge the Ops Portion of DevOps
0 likes2,842 views
The document discusses the integration of machine learning into DevOps practices, highlighting the need for operations to keep pace with development speed. It outlines the benefits of applying machine learning to IT operations and security to enhance efficiency and predictive capabilities. Key takeaways emphasize the importance of unified data and matured machine learning for effective analysis in DevOps environments.
Machine Learning to Turbo-Charge the Ops Portion of DevOps
- 1. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. Machine Learning Turbo-Charges the Ops Portion of DevOps DevOps.com webinar Tania Le Voi Director Oracle Management Cloud October, 2017
- 2. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
- 3. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Program Agenda Defining terms Why (Dev)Ops is Perfect for Machine Learning Making Machine Learning Smarter Q&A 1 2 3 4 #MgmtCloud or #DevOps or #devopswebinars @OracleMgmtCloud
- 4. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Program Agenda Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 1 2 3 4
- 5. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Defining Terms (source: wikipedia.com) • Machine Learning – Machine learning is the subfield of computer science that gives computers the ability to learn without being explicitly programmed. Evolved from the study of pattern recognition and computational learning theory in artificial intelligence, machine learning explores the study and construction of algorithms that can learn from and make predictions on data. • DevOps – DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. • Systems Management or IT Operations Management – IT Operations is responsible for the smooth functioning of the infrastructure and operational environments that support application deployment to internal and external customers, including the network infrastructure; server and device management; computer operations; IT infrastructure library (ITIL) management; and help desk services for an organization.
- 6. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Program Agenda Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 1 2 3 4
- 7. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | • Development is creating faster… ≥ Low-code ≥ Agile ≥ Microservices ≥ CI • (Dev)Ops is promoting faster… ≥ Containers ≥ IaaS & PaaS ≥ CD ≥ Packages • (the rest of)Ops is not moving any faster… ≥ #(*^(#^#)&^$(@^@($^ $(@)%&^$^**&^)!!!! ≥ #(*^(#^#)&^$(@^@($^ $(@)%&^$^**&^)!!!! ≥ … We have a problem: Dev has outpaced Ops
- 8. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | OPTION 1: Your Changes Don’t Hit Production Until Ops is Ready Confidential – Oracle Internal/Restricted/Highly Restricted 8 Option 2: You Promote Unmanaged Code Anyway One of Two Likely Outcomes, Both Bad
- 9. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | It's not my machines, it's your code! It's not my code, it's your machines! Where’s the data? 9 What does the data mean? The Reason: Ops Depends on Human Effort END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts END USER EXPERIENCE APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER VM CONTAINER INFRASTRUCTURE TIER VM CONTAINER Real Users Synthetic Users App metrics Transactions Server metrics Diagnostics Logs Host metrics VM metrics Container metrics CMDB Tickets Alerts
- 10. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. 10 Oracle Management Cloud INTELLIGENT, UNIFIED PLATFORM POWERED BY MACHINE LEARNING INFORMED BY A COMPLETE DATA SET HETEROGENEOUS AND OPEN APPLICATION MIDDLE TIER DATA TIER VIRTUALIZATION TIER INFRASTRUCTURE TIER END USER EXPERIENCE / ACTIVITY Unified Platform Global threat feeds Cloud access Identity Real users Synthetic users App metrics Transactions Server metrics Diagnostics logs Host metrics VM metrics Container metrics Configuration Compliance Tickets & Alerts Security & Network events
- 11. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | 11 Ops Data is Perfect for Machine Learning ✓Massive volume ✓Highly patterned ✓Predictable format ✓Silos can be unified ✓Seasonal trends ✓Known sources
- 12. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Algorithmic Approaches to IT Ops Data ANOMALY DETECTION CLUSTERING PREDICTION CORRELATION 12
- 13. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Program Agenda Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smart for IT Ops Q&A 1 2 3 4
- 14. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | ML is not smart out of the box for every question To make ML smarter, know the questions you want to ask, then… 1. Enhance Algorithms 2. Increase Breadth Maturing Machine Learning: A Three-Step Approach
- 15. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Increase Breadth With Data Unification & Normalization Oracle Management Cloud Data Store Log Analytics IT Analytics Infrastructure Monitoring ComplianceOrchestration Security Monitoring & Analytics Application Performance Monitoring Convert to Time Series (Clustering & Rollup) Base Lining & Anomaly Detection • Norm is repo by repo projects: slow and incremental. • By centralizing data, we are able to deliver ML driven features more quickly.
- 16. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Increase Depth With Context, Topology, & Domain Expertise Context: Forecasted SLA violation & observe divergent correlation. Topology: Tells us where to look. Domain Expertise: Allows to identify root cause.
- 17. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Applied Machine Learning for IT Operation Management 1. Metric data is automatically baselined and logs records are enriched 2. ITOM specific algorithms and models are provided out of the box. User input can be provided to further tune the algorithms. 3. Operationalization and automation using event processing for notifications and remedial orchestration actions Oracle Confidential – Internal/Restricted/Highly Restricted17 • Is increased load anomalous or expected at this time of the day • Early warning for future outages or SLA violations • Abnormal and rare system behavior • Capacity planning , WhatIf Analysis • All metric and log data is continually used to train the models • Corrective Actions: automated scale out, system restart, trigger diagnostic dump, revert configuration changes • Notification: Send alerts/notifications through a variety of channels • Incident: Create incident in 3rd party ticketing system, update status, attach evidence
- 18. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | APM AJAX Calls Anomalies • Ajax call metrics baselined and anomalies identified • Alert rules and corrective actions can be taken for anomalies
- 19. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Anomaly-based Alerts Alert when CPU Utilization(%) is anomalous Confidential – Oracle Internal/Restricted/Highly Restricted 19 Leverage Machine Learning
- 20. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Alert Notifications Confidential – Oracle Internal/Restricted/Highly Restricted 20
- 21. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Applied Machine Learning for Security and Compliance 1. Security log events are auto-enriched with user, asset, and threat intelligence context 2. Threat specific algorithms and security models are provided out of the box 3. Operationalization and automation using event processing for notifications and remedial SOC playbook execution Oracle Confidential – Internal/Restricted/Highly Restricted21 • Is the user privileged? • Is the asset regulated? • Is an accessed URL malicious? • Users coming from anomalous IPs • Users executing anomalous SQL queries • Assets with anomalous configuration drift • Identity: password reset, multi-factor authentication, privilege change • Asset: AV scan, endpoint data collection, configuration change • Incident: Create incident in 3rd party ticketing system, update status, attach evidence
- 22. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | • Common attributes in user actions are automatically and individually baselined from enriched security activity logs • Anomalies from self and/or peer group activity baselines per attribute are automatically flagged using prebuilt machine learning models • Behavioral threat specific remediation jobs are available for automatic or SOC analyst guided execution – URL blocking, identity actions, host isolation, endpoint data collection, incident management, firewall rule updates etc. 22 Example: Suspicious User Activity • Alice is executing actions from an IP address anomalous to her source IP baseline. • Bob is accessing internal assets anomalous for the sales team that he is part of. • Neil is accessing critical assets at a time of day that neither he nor his peer group are not normally active.
- 23. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | DEMO: Matured Machine Learning in Action
- 24. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. | Key Takeaways • DevOps depends on “Ops” speed matching “Dev” speed • The DevOps problem is well-suited to machine learning BUT… • Machine Learning must be matured • Unified data and context increases the effectiveness of ML and analysis Confidential – Oracle Internal/Restricted/Highly Restricted 24
- 25. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. Program Agenda Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 1 2 3 4
- 26. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. TRY IT FREE FOR 30 DAYS cloud.oracle.com/tryit oracle.com/managementcloud
- 27. Copyright © 2017,Oracle and/orits affiliates. All rights reserved. Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.