Machine Learning + Analytics in Splunk
7 likes4,031 views
This document contains a disclaimer stating that any forward-looking statements made during the presentation are based on current expectations and estimates and could differ materially. It also states that the information provided about product roadmaps is for informational purposes only and may change. The document provides an overview of machine learning, including definitions of common machine learning techniques like supervised learning, unsupervised learning, and reinforcement learning. It also describes Splunk's machine learning capabilities, including search commands, the Machine Learning Toolkit, and packaged solutions like Splunk IT Service Intelligence that incorporate machine learning.
![9
How Machines Learn
[Prediction]
• When we see thick clouds and an overcast sky, we
predict that it’s likely going to rain
[Estimation/ Regression]
• Estimate how much an apartment costs based on its
location, condition and prices of properties in that
neighborhood
[Classification/ Clustering]
• Determine the gender of a person based on her/his
features, hair style and the way s/he dresses
[Anomaly Detection]
• Identify the odd one out
[Reinforcement Learning]
• If I made a mistake this time, can I do better next time?
All of us have had some
experience in learning. But…
what’s behind our experience?
How do we translate that
knowledge to code?](https://image.slidesharecdn.com/splunklivemltk2-161110051840/85/Machine-Learning-Analytics-in-Splunk-9-320.jpg)
Machine Learning + Analytics in Splunk
- 4. 4 Humans are good at learning, but we get lost in volume and details…
- 5. 5 Why do we need Machine Learning? - Improve Decision Making - Forecast or Predict KPIs - Alert on Deviation - Uncover hidden trends or relationships All of this requires Diverse Data from across Many Silos. Lots of Unstructured, Real Time Data.
- 6. 6 Run the Business in Real-time Data From the Past Real-time Data Statistical Forecast T – a few days T + a few days Security Operations Center IT Operations Center Business Operations Center Predictive (Models) Descriptive (BI Tools, Data Lakes) Grey space
- 8. 8 ML 101: What is Machine Learning? What: “Field of study that gives computers the ability to learn without being explicitly programmed” – A. Samuel, 1959 How: Generalizing (learning) from examples (data) Simple ML workflow: – EXPLORE data – FIT models based on data – APPLY models in production – VALIDATE models – REPEAT
- 9. 9 How Machines Learn [Prediction] • When we see thick clouds and an overcast sky, we predict that it’s likely going to rain [Estimation/ Regression] • Estimate how much an apartment costs based on its location, condition and prices of properties in that neighborhood [Classification/ Clustering] • Determine the gender of a person based on her/his features, hair style and the way s/he dresses [Anomaly Detection] • Identify the odd one out [Reinforcement Learning] • If I made a mistake this time, can I do better next time? All of us have had some experience in learning. But… what’s behind our experience? How do we translate that knowledge to code?
- 10. 10 Major Types of Machine Learning 1. Supervised Learning: generalizing from labeled data
- 11. 11 Major Types of Machine Learning 2. Unsupervised Learning: generalizing from unlabeled data ?
- 12. 12 3. Reinforcement Learning: • System is rewarded (or punished) based on the outcomes it generates • Action leads to a change in the state of the world and generates an error score Major Types of Machine Learning
- 17. 17 Splunk Machine Learning Toolkit Assistants: Guide model building, testing, & deploying for common objectives Showcases: Interactive examples for typical IT, security, business, IoT use cases Algorithms: 25+ standard algorithms available prepackaged with the toolkit SPL ML Commands: New commands to fit, test and operationalize models Python for Scientific Computing Library: 300+ open source algorithms available for use Build custom analytics for any use case Extends Splunk platform functions and provides a guided modeling environment
- 20. 20 Summary: The ML Process Problem: <Stuff in the world> causes big time & money expense. Value Hypothesis Solution: Build ML model to forecast <possible incidents>, act pre-emptively & learn Operationalize 1. Get all the relevant data to the problem; Explore the data 2. Select and Fit an algorithm on the data, generating a model 3. Apply & Validate models until predictions solve the problem 4. Surface the model to X Ops, who consume the model to solve the problem
- 25. 25 Splunk: Data Fabric 2 OT Industrial Assets IT Consumer and Mobile Devices Real Time IT users Analysts Business Users Ad Hoc Search Custom Dashboards Monitor and Alert Reports/ Analyze Clickstreams HadoopDevices Networks GPS/ Cellular Online Shopping Carts Servers Applications Analysts Business Users Data Warehouses Structured Data Sources CRM ERP HR Billing Product Finance DB Connect Look-ups ODBC SDK API
- 28. 28 ML Is All Around You! Recall: EXPLORE > FIT > APPLY > VALIDATE > REPEAT • Face detection: find faces in images • Spam filtering: identify SPAM messages • Shopping Recommendations: predict what customers would like to buy • Fraud detection: identify credit card transactions which may be fraudulent in nature • Weather forecast: predict whether or not it will rain tomorrow; estimate daily max/min
- 32. 32 Reliable website updates Proactive website monitoring leads to reduced downtime 3 “Splunk ML helps us rapidly improve end-user experience by ranking issue severity which helps us determine root causes faster thus reducing MTTR and improving SLA” • Very frequent code and config updates (1000+ daily) can cause site issues • Find errors in server pools, then prioritize actions and predict root cause • Custom outlier detection built using ML Toolkit Outlier assistant • Built by Splunk Architect with no Data Science background ML Use Case Technical overview
- 34. 34 Next Steps with Splunk ML • Reach out to your Tech Team! We can help architect ML workflows. • Lots of ML commands in Core Splunk (predict, anomalydetection, stats) • ML Toolkit & Showcase – available and free, ready to use • Splunk ITSI: Applied ML for ITOA use cases – Manage 1000s of KPIs & alerts – Adaptive Thresholding & Anomaly Detection • Splunk UBA: Applied ML for Security – Unsupervised learning of Users & Entities – Surfaces Anomalies & Threats • ML Customer Advisory Program: – Connect with Product & Engineering teams - mlprogram@splunk.com
- 35. 35 What Else? • Get the Machine Learning Toolkit from Splunkbase • Go watch Machine Learning Videos on Splunk Youtube Channel http://tiny.cc/splunkmlvideos • Go watch the Machine Learnings talks from Conf 2016: – Advanced Machine Learning in SPL with the Machine Learning Toolkit by Jacob Leverich – Extending SPL with Custom Search Commands and the Splunk SDK for Python by Jacob Leverich • Early Adopter And Customer Advisory Program : mlprogram@splunk.com • Field ML Architects : Andrew Stein (astein@), Brian Nash (bnash@)
- 36. 36 Mark Your Calendars! • .conf2017 is going to DC! • Sept 26-28, 2017 • Walter E Washington Convention Center