Splunk for DataScience (.conf2014)
1 like624 views
This document provides a summary of using Splunk for data science. Splunk can be used for tasks like trend forecasting, anomaly detection, sentiment analysis, and market segmentation. It integrates data from various sources and allows querying and visualizing data. Splunk complements other data science tools by executing scripts from R and Python. Effective data visualizations are also important for communicating insights from data.
Splunk for DataScience (.conf2014)
- 1. 1 Copyright © 2014 Splunk Inc. Tom LaGatta Data Scientist, Splunk Olivier de Garrigues Sr Prof Services Consultant, Splunk Splunk for Data Science
- 2. 2 Disclaimer During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality describedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
- 3. 3 3 Key Takeaways Splunk is great for doing Data Science! Splunk complements other tools in the Data Science toolkit. Data Science is about extrac:ng ac:onable insights from data. 1 2 3
- 4. 4 About Us • Tom LaGatta, Data Scientist – Tom joined Splunk in Spring 2014 as a Data Scientist specializing in Probability and Statistics. Tom is an expert on the mathematics of inference, and he enjoys functional programming in languages like Clojure, Haskell & R.At Splunk, Tom is helping to develop our internal and external Data Science program and curriculum. Tom has a PhD in Mathematics from the University of Arizona, and until recently was a Courant Instructor at the Courant Institute at New York University.TomisbasedinNewYorkCity. • Olivier de Garrigues, Senior Professional Services Consultant – Olivier is based in London on the EMEA Professional Services team and has helped out more than 40 customers in 10 countries on variousSplunkprojectsinthepastyearandahalf.Priortothis,heworkedasaquantitativeanalystwith extensive use of MATLAB and R. He developed a keen interest in machine learning and enjoys dreaming about how to make Splunk better for data scientists, and helped develop the R ProjectApp. OlivierholdsanMSinMathematicsofFinancefromColumbiaUniversity.
- 5. 5 Splunk for Data Science
- 6. 6 What is Data Science? Data Science is about extracting actionable insights from data. • Helps people make better decisions. • Can be used for automated decision-making. • Data Science is cross-functional, and blends techniques & theories from: – CS / Programming – Math and Statistics – Machine Learning – Data Mining / Databases – Data Visualization • Don’t be afraid of Data Science! – Substantive / Domain Expertise – Social Science – Communication and Presentation – Accounting, Finance and KPIs – BusinessAnalytics
- 7. 7 Data Science & Analytics Teams There is no “one size fits all” data scientist. Data Science &Analytics teams are made up of people with complementary skill sets. Source: Schutt & O’Neil. Doing Data Science. 2013
- 8. 8 Splunk for Data Science Splunk is great for doing Data Science! • Integrate, query & visualize all the data: – Platform for machine data – Connects with any other data source • Easy-to-useAnalytics capabilities. • Powerful algorithms out-of-the-box. • Sharp visualizations and dashboards. • Deliver results to both IT & Business users. • Complements other Data Science tools (next slide).
- 9. 9 Splunk and Data Science Tools Splunk complements other tools in the Data Science toolkit: • Hadoop: the workhorse of the Data Science world. Using Hunk, you can integrate Hadoop & HDFS seamlessly into Splunk. • R & Python: the preferred languages of Data Science. Execute R & Python scripts in your Splunk queries using the R ProjectApp & SDK for Python. • SQL& other RDBMS: valuable stores for customer & product data. Use Splunk’s DB ConnectApp to mash relational data up with machine data. • External tools: export finalized data from Splunk using the ODBC Driver. – Tip: do all your data processing in Splunk/Hunk, and export only the final results. • D3 Custom Visualizations: sharp dashboards & reports using Splunk.
- 10. 10 Splunk and Data Science Use Cases Green Use Cases (easy out of the box) Yellow Use Cases (needs tinkering) Trend Forecasting D3 Custom Visualizations A/B Testing Predictive Modeling Root Cause Analysis Sentiment Analysis Anomaly Detection Conversion Funnel/Pathing Market Segmentation More Algorithms via R & Python Topic Modeling Capacity Planning Correlate Data from 2+ Sources Data Munging & Normalization KPIs & Executive Dashboards Splunk is a powerful tool for lots of Data Science use cases:
- 11. 11 Data Science Use Cases
- 12. 12 Use Case: Trend Forecasting Trend Forecasting: Given past & realtime data, predict future values & events. • Common applications: – Forecast revenue & other KPIs – Web server traffic & product downloads – Customer conversion rates – Estimate MTTR & server outages – Resource & capacity planning (AWSApp) – Security threats (Enterprise SecurityApp) • The “true” course of events can (and will) take only one of many divergent paths. But which one…? • Be mindful of rare events & black swans!
- 13. 13 Splunk Solution: predict! predict command: forecast future trajectories of time series. • Implements a Kalman filter to identify seasonal trends. • Gives an “uncertainty envelope” as a buffer around the trend. • Tip:Always run the predict command on LOTS of past data. Capture low-frequency and high-frequency trends. • Remember: the future is always uncertain…
- 14. 14 Splunk Solution: Predict App David Carasso’s PredictApp: forecast future values of individual events. – 8 minute walkthrough: https://www.youtube.com/watch?v=ROvaqJigNFg • Implements a Naïve Bayes classifier. • You have to train models! • Train a model to predict any target field using any reference field(s): fields ref1, ref2, ..., target | train my_model from target! • Guess target field for incoming events: guess my_model into target • Temporal or non-temporal prediction (include _time among reference fields).
- 15. 15 Concept: Supervised Learning & Classification Supervised learning: use observed training data to classify values of unknown testing data. • predict command (Kalman filter): Training data = timechart of past & realtime values. Testing data = time range for future values. • PredictApp (Naïve Bayes classifier): Training data = events with reference & target fields. Testing data = events with reference fields but not target field. • Tip: only deploy models & algorithms after extensive testing & evaluation. • More powerful learning algorithms using R ProjectApp or SDK for Python.
- 16. 16 Demo: Predict App • Train a model to predict movie Rating based on MovieID, UserID, Genre,Tag index=movielens Timestamp < 1199188800 UserID=593* | eval original_rating = case(Rating<3,"Dislike", Rating=3,"Neutral", Rating>3,"Like") | fields original_rating MovieID UserID Genre Tag | train rating_model from original_rating! • Guess Rating for test data based on trained model index=movielens Timestamp > 1199188800 UserID=593* | guess rating_model into guessed_rating | top original_rating guessed_rating! • Accuracy of model: correct on 97.6% of values. • Tip: always train on LOTS of training data. • Evaluate before deploying.
- 17. 17 Use Case: Sentiment Analysis SentimentAnalysis: the assignment of “emotional” labels to textual data. • Can be simple +1 vs. -1, or more sophisticated: “happy”, “angry”, “sad”, etc. • Analyze tweets, emails, news articles, logs or any other textual data! – Social data correlates with other factors. • Typically done via supervised learning: – Train a model on labeled corpus of text. – Test the model on incoming text data. • Read more about SentimentAnalysis: – Chapter14ofBigDataAnalyticsUsingSplunk(pp.255-282). – MichaelWilde&DavidCarasso.SocialMedia&SentimentAnalysis..conf2012 3rd 8th 4th 1st 2nd 2011 Irish General Election 17% 1.8% 10% 36% 19% ★ r=.79
- 18. 18 Splunk Solution: Sentiment Analysis App David Carasso’s SentimentAnalysisApp assigns binary sentiment values to textual data (logs, tweets, email, etc.). • Naïve Bayes classifier under the hood. • Twitter & IMDB models out of the box. • Can guess language of authorship, and “heat”, a measure of emotional charge. • Tip: compare relative sentiment changes across time & groups. • How to train your own models: http://answers.splunk.com/answers/59743
- 19. 19 Demo: Sentiment Analysis App
- 20. 20 Use Case: Anomaly Detection • An anomaly (or outlier) is an event which is vastly dissimilar to other events. • Anomaly Detection is one of Splunk’s most common use cases. Examples: – Transactions which occur faster than humanly possible. – DDoS attacks from IPaddress ranges. – High-value customer purchase patterns. • Quick techniques for finding statistical outliers: – Non-average outliers: more than 2*stdev from the avg. – Non-typical outliers: more than 1.5*IQR above perc75 or below perc25. • Tip: save these as eventtypes for automated outlier detection. • Once anomalies have been found, dig deeper to discover root causes.
- 21. 21 Splunk Solution: cluster • Anomalies are dissimilar to other events (by definition). • We can use clustering algorithms to help us detect anomalies: – Non-anomalous events typically form a few large clusters. – Anomalous events typically form lots of small clusters. • Cluster your data, sort ascending: cluster showcount=true labelonly=true | sort cluster_count cluster_label! • Remember: there is no “right way” to find all anomalies. Explore your data!
- 22. 22 Concept: Unsupervised Learning & Clustering • Aclustering algorithm is any process which groups together similar things (events, people, etc), and separates dissimilar things (events, people, etc). • Clustering is unsupervised: choose labels based on patterns in the data. • Clustering is in the eye of the beholder: – Lots of different clustering algorithms. – Lots of different similarity functions. • Do not confuse with: – Computer cluster: a group of computers working together as a single system. – Splunk cluster: a group of Splunk indexers replicating indexes & external data.
- 24. 24 Splunk Solution: Other Commands • anomalies: – Assigns an “unexpectedness” score to each event. • anomalousvalue: – Assigns an “anomaly score” to events with anomalous values. • outlier: – Removes or truncates outliers. • kmeans: – Powerful clustering algorithm. You choose k = # of clusters.
- 25. 25 Splunk Solution: Prelert (Partner App) • ManagesAnomaly Detection directly. – Pre-built dashboards, alerts,API. – Use cases: Security, ITOps /APM, DevOps – Godfrey Sullivan: "beautifully adjacent and complimentary to what Splunk does” • Can download from SplunkApps. – May save you time withAnomaly Detection. – Can also be good source of inspiration for your ownAnomaly Detection dashboards. • Keep in mind Prelert is a paid app: – Cost: $225/month @ 5GB
- 26. 26 Use Case: Market Segmentation • Market Segmentation: group customers according to common needs and priorities, and develop strategies to target them. – Market segments are internally homogeneous, and externally heterogeneous. i.e., market segments are clusters of customers. • Many reasons for Market Segmentation: – Different market segments require different strategies. – Customers in same segment have similar product preferences. Different segments, different preferences. – Segments should be reasonably stable, to allow for historical analysis (good for Data Science). • Use Splunk’s clustering algorithms to identify and label market segments!
- 28. 28 Intro to Data Visualization • Data Visualization is the creation and study of the visual representation of data, and is a vital part of Data Science. • The goal of data visualization is to communicate information: – Visualizations communicate complex ideas with clarity, precision, and efficiency. – Transmission speed of the optic nerve is about 9Mb/sec – fast image processing. – Pattern matching, edge detection. – Visualizations pack lots of information into small spaces. More than text alone!
- 29. 29 Telling Stories with Data Visualizations • We process data in linear narratives: even dashboards go top-to-bottom. • Visualizations help pierce the monotony of text, number & data streams. • Think about the story you’re telling: – Empathize with the viewer. – What’s their takeaway? • Agood visualization tells its own story: “Island Nation Obtains Favourable Balance ofTrade; Goes OnTo RuleThe World.” • Weave multiple visualizations together to tell more effective stories. William Playfair (1786)
- 30. 30 Source: New York Times. May 17, 2012 Splunk
- 31. 31 Source: New York Times. May 17, 2012 Splunk
- 32. 32 Tips for Effective Data Visualizations • #1 tip: Plot the most important keys on x & y axes. – You choose “most important.” – You might need >1 visualization. • Manipulate size, color and shape to convey additional information. • Annotate, label and add icons ✔︎ • Use chart overlay to correlate data sources. Mix histograms & line charts ↑↑↑ • Manipulate numerical scale: linear vs. log scales (previous 2 slides). • Read more about Data Visualization: – Tableau’s whitepaper, VisualAnalysis Best Practices (2013). – EdwardTufte’s The Visual Display of Quantitative Information (2001).
- 33. 33 • Splunk now supports D3 visualizations with some minor customization. • Satoshi’s talk: “I want that cool viz in Splunk!” • Resources for Custom Visualizations: – Splunk Web FrameworkToolkit https://apps.splunk.com/app/1613/ – Splunk 6.x Dashboard Examples https://apps.splunk.com/app/1603/ – Custom SimpleXMLExtensions http://apps.splunk.com/app/1772/ – Lots more D3 visualizations for use: https://github.com/mbostock/d3/wiki/Gallery D3 Custom Visualizations in Splunk
- 35. 35 How-to for Sankey Charts • Install the Custom SimpleXMLExtensions app: http://apps.splunk.com/app/1772/ • Create your own app, and install Sankey chart components: – Drop autodiscover.js in $SPLUNK_HOME/etc/apps/<YOURAPP>/appserver/static – Copy & paste /sankeychart/ subfolder into $SPLUNK_HOME/etc/apps/<YOURAPP>/ appserver/static/components – Restart Splunk. • In your dashboard: – Include script="autodiscover.js" in <form> or <dashboard> opening tag – Insert XMLsnippet from 2- or 3-node Sankey dashboard example – Change 2 instances of “custom_simplexml_extensions” to <YOURAPP>. – Update search and “data-options” parameters (nodes) in XMLto reflect your data.
- 36. 36 Know Your Audience • Finally, keep in mind your audience: who are they, what questions do they care about, and how do they want to consume the data? – Executive: KPIs, charts, tables with icons ✔︎ – MarketingAnalyst: KPIs & metrics. Sharp images for their own reports & decks.Tableau. – Data Scientist: output clean data to organized data stores (Hunk, HDFS, SQL, NoSQL). – Sysadmin: sparklines, gauges for activity & MTTR, tables with highlighted anomalies. – Security Ops: maps with detailed overlays, drill down on anomalous events. • Bring it back to the business problem & use case!
- 37. 37 3 Key Takeaways Splunk is great for doing Data Science! Splunk complements other tools in the Data Science toolkit. Data Science is about extrac:ng ac:onable insights from data. 1 2 3
- 38. 38 List of References Good books on Data Science: • Schutt & O’Neil. Doing Data Science. O’Reilly 2013 • Provost & Fawcett. Data Science for Business. O’Reilly 2013 • Max Shron. Thinking With Data. O’Reilly 2014 • Edward Tufte. The Visual Display of Quantitative Information. Graphics Press 2001 • Zumel & Mount. Practical Data Science with R. Manning 2014 • Hastie et al. Elements of Statistical Learning. Springer-Verlag 2009 (free PDF!) Using Splunk for Data Science: • Zadrozny, Kodali (and Stout). Big Data Analytics Using Splunk. Apress 2013 • David Carasso. Exploring Splunk. CITO Research 2012 • David Carasso. Data Mining with Splunk. .conf2012 • Michael Wilde & David Carasso. Social Media & Sentiment Analysis. .conf2012 Good free references: • Tableau. Visual Analysis Best Practices. Tableau 2013 • King & Magoulas. 2013 Data Science Salary Survey. O’Reilly 2013 • DJ Patil. Building Data Science Teams. O’Reilly 2013 • Cathy O’Neil. On Being A Data Skeptic. O’Reilly 2013