Splunk Artificial Intelligence & Machine Learning Webinar
4 likes2,886 views
The Splunk webinar on artificial intelligence and machine learning showcased the company's offerings in these areas, including the Machine Learning Toolkit (MLTK) and customer success stories like those from T-Mobile and Israel's Ministry of Energy. It emphasized the importance of data science expertise and domain knowledge for effective machine learning application. Participants were invited to engage with the content through live demos and Q&A sessions.
Splunk Artificial Intelligence & Machine Learning Webinar
- 1. © 2020 SPLUNK INC. Splunk Artificial Intelligence & Machine Learning Webinar April 1, 2020 Philipp Drieger | Principal Machine Learning Architect
- 2. © 2020 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved. Forward-Looking Statements
- 3. © 2019 SPLUNK INC. • | where _time @ Splunk > 5y • Previous: • +15y in research, software development, visual arts • +3y SE across portfolio & domains in CEMEA & EE • Specializations • Anomaly Detection, Data Mining, NLP, Advanced Analytics and Visualizations • Applied Data Science, Machine Learning, Graph Theory and Network Science • GPU Computing, Deep Learning • Role @ Splunk • Principal Machine Learning Architect • Author of DGA App for Splunk • Author of MLTK Container for Splunk • Author of Deep Learning Toolkit for Splunk • Blog posts, conf talks, hackathons etc. • Ensure Customer and Partner Success with ML Philipp Drieger
- 4. © 2019 SPLUNK INC. Webinar Housekeeping
- 5. © 2020 SPLUNK INC. Webinar Housekeeping Rules You will be muted by default to ensure a background noise free presentation for everyone My fellow colleagues will help instantly with questions on the chat or help to consolidate We will have a look at open questions on the fly and answer as soon as possible in breaks or at the end of the session Please stay muted, but … … feel free to ask questions in the chat anytime … … and allow for a bit time to answer Thanks for contributing to make this webinar a positive experience for everyone!
- 6. © 2019 SPLUNK INC. Agenda ● Webinar Housekeeping ● Introduction to AI and ML Features in Splunk ● Customer Use Case Examples ● Live Demo of Machine Learning Toolkit, with examples for: ○ Methods for Anomaly Detection ○ Predictive Analytics and Forecasting ○ Clustering ● Custom Machine Learning, incl.: ○ Advanced Containerization ○ Expansion with MLSPL API ● Open Q&A
- 7. © 2019 SPLUNK INC. Intro
- 8. © 2020 SPLUNK INC. Our World Never Stops Evolving. New Ideas. New Devices. New Processes. © 2019 SPLUNK INC.
- 9. © 2020 SPLUNK INC. Every Company Has a Universe of Real-time Data Creating More Opportunities and Threats than Ever Before New Data Streams & Devices New Apps & App Logs Financial Account & Operating Systems Database Logs Network Logs New Technolo gy ATM Sensor Data Transacti on Data Proxy Data Firewall Logs © 2019 SPLUNK INC.
- 10. © 2020 SPLUNK INC. Turning Real-time Data Into Action is Hard Data Lakes Master Data Management ETL Point Data Management Solutions Data Silos © 2019 SPLUNK INC.
- 11. © 2020 SPLUNK INC. Data LakesMaster Data ManagementETL Point Data Management Solutions Data Silos IT Security IoT Biz Analytics The Data-to-Everything Platform © 2019 SPLUNK INC.
- 12. © 2020 SPLUNK INC. Data LakesMaster Data ManagementETL Point Data Management Solutions Data Silos Any Structure Any Source Any Time Scale AC T INVESTIGAT E ANALYZE MONITO R IT Security IoT Biz Analytics © 2019 SPLUNK INC.
- 13. © 2020 SPLUNK INC. Splunk: The Data-to-Everything Platform Bring data to every question, decision and action Cloud Monitoring Application Lifecycle Analytics Application Release Analytics Container Monitoring Infrastructure Monitoring Advanced Threat Detection Insider Threats Incident Investigation and Forensics SOC Automation Compliance Real-Time Monitoring and Diagnostics ICS Security Predictive Analytics Facilities Management Business Process Mining Customer Experience Optimization Incident Management Digital Marketing Optimization IoT Biz AnalyticsIT Security
- 14. © 2019 SPLUNK INC. Intro AI | ML | DL
- 15. © 2019 SPLUNK INC. “Humans are good at Learning… but we get lost in volume and detail.”
- 16. © 2020 SPLUNK INC. AI, ML, DL “A Function that maps features to an output” = AI “A Function that learns patterns in your data without being explicitly programmed” = ML Types of ML Supervised Unsupervised Reinforcement Lots of opinions exist. Myths as well…
- 17. © 2020 SPLUNK INC. What ML & AI are not Machine Learning is not MagicAI Bu zzGarbage Data = Useless Predictions • Data Scientists spend 80% of their time cleaning, munging and collecting data • Throwing more data at an algorithm will not result in solving all of your SOC issues • Machine Learning requires a solid understanding of statistics and the scientific method ML & AI require you to understand the fundamental business problem you want to solve.
- 18. © 2020 SPLUNK INC. What ML & AI are not Machine Learning is not Magic ML is not a replacement for expert analysts, or engineers. ML requires Subject Matter Experts to enhance security & IT operations. Analysts are required to provide feedback to the models to adjust thresholding rules and reduce false positives. AI Bu zz
- 19. © 2020 SPLUNK INC. Problem: DGA domains are computer generated pseudo-random character strings used by attackers, blacklisting an infinite number of domains is not feasible. Hypothesis: “Are there patterns in domain generation algorithms that can be exploited to identify newly generated domains as threats in real-time?” Example Domains: Machine Learning & AI What does the scientific method look like in the IT & Security Space? http://87hfdredwertyfdvvlkgdrsadm.net/af/GHFbfsalku65 http://87hfdredwertyfdvvlkgdrsadm.net/af/sdgLKJvgh http://wszystkodokuchni.pl/34f43
- 20. © 2019 SPLUNK INC.
- 21. © 2019 SPLUNK INC. Machine Learning Tour
- 22. © 2020 SPLUNK INC. What Data Scientists Really Do Data Preparation accounts for about 80% of the work of data scientists “Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says”, Forbes Mar 23, 2016
- 23. © 2020 SPLUNK INC. Splunk Customers Want Answers from their Data ► Deviation from past behavior ► Deviation from peers ► (aka Multivariate AD or Cohesive AD) ► Unusual change in features ► Identify peer groups ► Event Correlation ► Reduce alert noise ► Behavioral Analytics Anomaly detection Predictive Analytics Clustering ► Predict Service Health Score/Churn ► Predicting Events ► Trend Forecasting ► Detecting influencing entities ► Early warning of failure
- 24. © 2020 SPLUNK INC. Skill Areas for Machine Learning @ Splunk Domain Expertise (IT, Security…) Data Science Expertise Splunk Expertise MLTK Splunk ML Toolkit facilitates and simplifies via examples & guidance Premium solutions provide out of the box ML capabilities. ITSI, UBA • Statistics/math background • Algorithm selection • Model building • Identify use cases • Drive decisions • Understanding of business impact • Searching • Reporting • Alerting • Workflow
- 25. © 2020 SPLUNK INC. Overview of Machine Learning at Splunk CORE PLATFORM SEARCH + Smarter Splunk PACKAGED PREMIUM SOLUTIONS MACHINE LEARNING TOOLKIT Platform for Operational Intelligence
- 26. © 2020 SPLUNK INC. Splunk Machine Learning Toolkit (MLTK) Built for the Citizen Data Scientist • Experiments and Assistants: Guided model building, testing, and deployment for common objectives • Algorithms: 80+ standard algorithms (supervised & unsupervised) Extensible to operationalize any use case • Python for Scientific Computing Library: Access to 300+ open source algorithms • Deep Learning Toolkit : Supports NN and GPU accelerated machine learning • ML-SPL API: Import any open-source or proprietary algorithm Extends Splunk to operationalize Machine Learning
- 27. © 2020 SPLUNK INC. Quick Overview of Algorithms in MLTK https://www.splunk.com/pdfs/solution-guides/machine-learning-quick-ref-guide.pdf
- 28. © 2020 SPLUNK INC. Custom ML with the Splunk Platform Visualize & Share Clean & Munge Operationalize Monitor Alert Search & Explore Collect Data Build, Test, Improve Models Ecosystem MLTK Choose Algorithm Ecosystem Splunk Splunk Splunk Splunk MLTK Splunk Ecosystem Splunk Operationalized Data Science Pipeline Ecosystem MLTK Splunk Splunk’s App Ecosystem contains 1000’s of free add-ons for getting data in, applying structure and visualizing your data giving you faster time to value. The Machine Learning Toolkit delivers new SPL commands, custom visualizations, assistants, and examples to explore a variety of ml concepts. Splunk Enterprise is the mission-critical platform for indexing, searching, analyzing, alerting and visualizing machine data. Pre-processing Feature Selection MLTK Splunk MLTK Splunk Platform for Operational Intelligence
- 29. © 2019 SPLUNK INC. Customer Success Stories
- 30. © 2020 SPLUNK INC. Recent Customer Success Stories @ .conf19 Enhanced Anomaly Detection: Join T-Mobile and Splunk as we Deep Dive an Enterprise-IT Operational Use Case Add value to your SIEM: how Israel's Ministry of Energy applies Machine Learning to protect their Critical Infrastructure and OT Operations Augment Your Security Monitoring Use Cases with Splunk's Machine Learning Toolkit T-Mobile (US) Ministry of Energy, State of Israel SIEMENS AG Learn more at conf.splunk.com with over 900+ presentations available online!
- 31. © 2020 SPLUNK INC. Splunk ML Advisory Customers
- 32. © 2019 SPLUNK INC. 1) Get help from the Splunk Data Scientists to solve your business use case with Machine Learning Toolkit 2) Complimentary support with your Enterprise or Cloud license 3) Early access to new Machine Learning features 4) Results in opportunity to tell your success story with Splunk 5) Contact mlprogram@splunk.com for more information or your Splunk account team Splunk Machine Learning Advisory Program
- 33. © 2019 SPLUNK INC. Splunk’s Machine Learning Toolkit
- 34. © 2019 SPLUNK INC.
- 35. © 2019 SPLUNK INC. Machine Learning Toolkit 5.0 New capabilities continue to make machine learning easily accessible by more users and extensible with connectors • Easier to navigate with a new, modern showcase layout • Smarter with the introduction of the new Smart Outlier Detection Assistant for anomaly detection • Migration to Python 3 • Applicable to more use cases with the Smart Forecasting Assistant with Multivariate Forecasts and Special Days Effects
- 36. © 2019 SPLUNK INC. Deploying and Applying ML with Splunk
- 37. © 2020 SPLUNK INC. Continuous Data Ingest at Scale DevelopVisualize PredictAlertSearch Engineers Data Analysts Security Analysts Business Users Native Inputs TCP, UDP, Logs, Scripts, Wire, Mobile Industrial Data SCADA, AMI, Meter Reads Modular Inputs MQTT, AMQP, COAP, REST, JMS HTTP Event Collector Token Authenticated Events Technology Partnerships Kepware, AWS IoT, Cisco, Palo Alto Maintenance Info Asset Info Data Stores External Lookups/EnrichmentOT Industrial Assets IT Consumer and Mobile Devices Real Time
- 38. © 2020 SPLUNK INC. Every Search Can Use Machine Learning Search Third-Party Applications Smartphones and Devices Tickets Email Send an email File a ticket Send a text Flash lights Trigger process flow AlertReal Time OT Industrial Assets IT Consumer and Mobile Devices
- 39. © 2020 SPLUNK INC. MLTK + Python for Scientific Computing persisted model SearchReal Time Visualize Alert | fit y from x* into “model” | apply “model” … Python for Scientific Computing OT Industrial Assets IT Consumer and Mobile Devices
- 40. © 2020 SPLUNK INC. Deep Learning Toolkit for Splunk persisted model SearchReal Time Visualize Alert | fit y from x* into “model” | apply “model” … OT Industrial Assets IT Consumer and Mobile Devices
- 41. © 2019 SPLUNK INC. Live Demo Splunk Machine Learning Toolkit (MLTK)
- 42. © 2019 SPLUNK INC. Custom ML and Advanced Containerization
- 43. Announcing the Deep Learning Toolkit for Splunk with TensorFlow 2.0, PyTorch, NLP and Jupyter Lab Notebooks
- 44. © 2020 SPLUNK INC. Seamlessly Integrate with Splunk Enterprise and Machine Learning Toolkit Workflows Freedom of Code within Jupyter Lab Notebooks for Advanced Modelling with TensorFlow and PyTorch GPU accelerated Deep Learning for Compute Intensive Training Workloads Key Benefits of the MLTK Container
- 45. © 2020 SPLUNK INC.
- 46. © 2020 SPLUNK INC.
- 47. © 2020 SPLUNK INC.
- 48. © 2020 SPLUNK INC.
- 49. © 2020 SPLUNK INC.
- 50. © 2020 SPLUNK INC.
- 51. © 2020 SPLUNK INC.
- 52. © 2020 SPLUNK INC.
- 53. © 2020 SPLUNK INC.
- 54. © 2020 SPLUNK INC.
- 55. © 2019 SPLUNK INC. 1. Extend your Splunk platform with the Deep Learning Toolkit for Splunk 2. Integrate custom advanced deep learning and NLP models into Splunk using a predefined Jupyter Notebook workflow for rapid model development. 3. Leverage GPUs for compute intense training tasks Deep Learning Toolkit for Splunk Key Takeaways
- 56. © 2019 SPLUNK INC.
- 57. © 2019 SPLUNK INC. Extend MLTK using the MLSPL API Example
- 58. © 2019 SPLUNK INC. How to wrap a custom algo into MLTK? Good news: You can wrap any python based algo using the MLSPL API.
- 59. © 2020 SPLUNK INC. Documentation How to wrap a custom algo into MLTK? More examples: https://github.com/splunk/mltk-algo-contrib
- 60. © 2020 SPLUNK INC. Why not use Jupyter Notebooks from DLTK to rapidly develop the algorithm and port it back into MLTK ? Download and get started with DLTK today: https://splunkbase.splunk.com/app/4607/
- 61. © 2020 SPLUNK INC. This was pretty simple! We wrapped it in...
- 62. © 2019 SPLUNK INC. How to use it in Splunk?
- 63. © 2020 SPLUNK INC. Extract graph structures with SPL … | stats count by src dest
- 64. © 2020 SPLUNK INC. Algorithm Example: Graph Centrality
- 65. © 2020 SPLUNK INC.
- 66. © 2020 SPLUNK INC.
- 67. © 2020 SPLUNK INC.
- 68. © 2019 SPLUNK INC.
- 69. © 2019 SPLUNK INC. Outlook into Future: new products Data Stream Processor (DSP)
- 70. © 2020 SPLUNK INC. Splunk Data Stream Processor Log Files Online Shopping Cart Cell Phones and Devices RFID Messaging Patient Generated Data Servers Web Services Call Detail Records Protect sensitive data Take action on data in motion Turn raw data into high-value information Distribute data to Splunk or other destinations Filter Format Enrich Mask Sensitive Data Detect data patterns or conditions Aggregate Normalize Transform Track and monitor pipeline health Splunk Data Stream Processor A real-time stream processing solution that collects, processes and delivers data to Splunk and other destinations in milliseconds Data Warehouse Public Cloud Message Bus
- 71. © 2020 SPLUNK INC. Use Cases Filter out or route noisy data to specific destinations Data Routing Filtering/ Noise Removal Data Formatting Guarantee delivery of high-volume, high-velocity data to multiple destinations Format or organize data using various functions based on specified conditions Aggregate data based on specific conditions and identify abnormal patterns in data Data Aggregation DATA IN MOTION
- 72. © 2020 SPLUNK INC. Introducing Unbounded ML in DSP Streaming Analytics : Derive insights while data is still in motion ● Automatic Detection of patterns and anomalies in raw logs ● Advanced pattern matching ● Sequential Outlier detection ● Multi-source correlation Derive insights on data in motion Continuous Intelligence ● Algorithms that learn continuously ● No downtime machine learning systems ● Unbounded in cardinality of models and data volume Advanced Analytics ● Online classification, clustering, time series forecasting, changepoint detection etc baked in ● Self tuning algorithms, no manual hyper parameter tuning needed
- 73. © 2019 SPLUNK INC. Anomaly Detection on Stream. General Questions: DSP-SplunkNext@splunk.com
- 74. © 2019 SPLUNK INC. Open Q&A
- 75. © 2020 SPLUNK INC. Link Collection Apps: DGA App For Splunk Machine Learning Toolkit (MLTK) ML-SPL Performance App For Machine Learning Toolkit Deep Learning Toolkit for Splunk (DLTK) 3D Graph Network Topology Visualization MLTK github contribution Customer Use Cases from .conf archiv: .conf sessions with AI/ML related content Conf Customer Case T-mobile (Video + Slides) Conf Customer Case Siemens (Video + Slides) Conf Customer Case Israel Ministry of Energy (Slides) New Splunk Products: Splunk Data Stream Processor (DSP) with streaming ML Where can I learn more? Documentation & Blogs: ML Blogposts Blogpost Graph ML Machine Learning Quick Reference Guide Documentation Density Function ML-SPL API Guide Splunk ML related education courses: Splunk Training for Analytics and Data Science Upcoming Splunk 4 Rookies Sessions Splunk 4 Rookies English Splunk 4 Rookies German
- 76. Thank You © 2019 SPLUNK INC.