You've Got Junk In Your Splunk
2 likes470 views
The document discusses the challenges associated with machine-generated data in enterprises, highlighting its complexity and the difficulties in extracting value from it. It introduces Splunk as a tool that can analyze this data in real-time, enabling various case studies that demonstrate its effectiveness in business analytics and information security. Notable examples include 7/11's use of Splunk to optimize promotional planning and Northern Arizona University's proactive security measures.
![YOU’VE GOT JUNK IN YOUR SPLUNK - MACHINE DATA
WHAT DOES “MACHINE DATA” LOOK LIKE?
2015-10-17 13:08:51-0700 [SSHService ssh-userauth on HoneyPotTransport,
2323,93.158.203.167] login attempt [root/12345] succeeded
64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/
Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1"
401 12846
{"created_at":"Mon Sep 28 19:39:04 +0000 2015”,”user”:”yourbuddyconner",
"id":648582717068587000,"id_str":"648582717068587009","text":"The
amount of local news stations treating the Facebook outage as news is too
damn high. #FacebookDown #TwitterIsUp #Facebook”,"entities":{"hashtags":
[{"text":"FacebookDown","indices":[89,102]},{"text":"TwitterIsUp","indices":
[103,115]},{"text":"Facebook","indices":[116,125]}],"symbols":
[],"user_mentions":[],"urls":[]}}
message_id=53088 timestamp="2015-02-03 20:30:06" date_read="2015-02-03
20:29:20" is_from_me=1 is_read=1 handle=+9999999999 service=iMessage
message="I mean, I can, those pancakes were so good"
Honeypot Logs:
Webserver Logs:
Tweets:
Text Messages:
SERVICE NAME
USERNAME PASSWORD STATUS MESSAGE
IP ADDRESS
HTTP METHOD
TIMESTAMP TWITTER HANDLE
HASHTAGS
PHONE NUMBER
MESSAGE
TIMESTAMP](https://image.slidesharecdn.com/conner-swann-youve-got-junk-in-your-splunk-151120184055-lva1-app6892/85/You-ve-Got-Junk-In-Your-Splunk-8-320.jpg)
![YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE RESULTS
▸ Comparing incoming sentiment with outgoing sentiment
▸ index=text_messages is_from_me=0 | sentiment twitter message | eval
diff=sentiment-0.788400 | timechart avg(diff) as sentiment_from span=1mon | appendcols
[search index=text_messages is_from_me=1 | sentiment twitter message | eval
diff2=sentiment-0.788400 | timechart avg(diff2) as sentiment_me span=1mon]
▸ Conclusion: Outgoing sentiment is at times closely coupled with incoming
sentiment](https://image.slidesharecdn.com/conner-swann-youve-got-junk-in-your-splunk-151120184055-lva1-app6892/85/You-ve-Got-Junk-In-Your-Splunk-26-320.jpg)
You've Got Junk In Your Splunk
- 1. YOU’VE GOT JUNK IN YOUR SPLUNK Conner Swann NAU Information Technology Services
- 2. YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM WHAT IS THE PROBLEM? ▸ Most enterprise data is machine-generated ▸ Machine data is often-times not human readable ▸ Numerous disparate data sources and formats ▸ Different implementations and architectures ▸ Virtualized Applications ▸ 3rd Party Off-Site Solutions (“The Cloud”) ▸ On-Site Hardware
- 3. YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM SERIOUSLY? THIS IS A PROBLEM? ▸ Dan the developer is asked to help figure out why his code is crashing on Sundays at Midnight ▸ Sally the SysAdmin has no idea why users from one office location can’t log in to their computers ▸ Ivan the InfoSec Analyst has no idea a hacker in Bulgaria is sending spam from his servers ▸ Billy the Business Analyst needs to figure out what localities are using his company’s applications ▸ Molly the Marketing Executive needs to analyze her affiliate marketing campaigns to see if improvements can be made
- 4. YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM YES, IT’S A PROBLEM. ▸ Machine Data is the most rapidly growing and complex segment of “Big Data” ▸ It’s generated 24/7/365 by nearly every device in existence and will continue to be generated forever ▸ Contains categorical record of every activity and behavior ▸ Value from this data is largely untapped — extremely difficult to process and analyze in a timely manner by traditional means
- 5. YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA ▸ Business Application Data ▸ Relational Data, highly structured, inflexible schema ▸ Financial Records, multidimensional data, computationally intense at times ▸ Rare reports, never realtime
- 6. YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA ▸ Human Generated Data ▸ Created as a result of Human-Human interaction ▸ Email, IM, Voice, Text, Video ▸ Stored in central corporate data centers, on mobile devices and on individual PCs
- 7. YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA ▸ Machine Data ▸ Time series, diverse, unstructured, no predefined all- encompassing schema ▸ Encapsulates Human Generated Data ▸ Generated by all IT systems ▸ Absolutely ridiculous volume of data
- 8. YOU’VE GOT JUNK IN YOUR SPLUNK - MACHINE DATA WHAT DOES “MACHINE DATA” LOOK LIKE? 2015-10-17 13:08:51-0700 [SSHService ssh-userauth on HoneyPotTransport, 2323,93.158.203.167] login attempt [root/12345] succeeded 64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/ Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846 {"created_at":"Mon Sep 28 19:39:04 +0000 2015”,”user”:”yourbuddyconner", "id":648582717068587000,"id_str":"648582717068587009","text":"The amount of local news stations treating the Facebook outage as news is too damn high. #FacebookDown #TwitterIsUp #Facebook”,"entities":{"hashtags": [{"text":"FacebookDown","indices":[89,102]},{"text":"TwitterIsUp","indices": [103,115]},{"text":"Facebook","indices":[116,125]}],"symbols": [],"user_mentions":[],"urls":[]}} message_id=53088 timestamp="2015-02-03 20:30:06" date_read="2015-02-03 20:29:20" is_from_me=1 is_read=1 handle=+9999999999 service=iMessage message="I mean, I can, those pancakes were so good" Honeypot Logs: Webserver Logs: Tweets: Text Messages: SERVICE NAME USERNAME PASSWORD STATUS MESSAGE IP ADDRESS HTTP METHOD TIMESTAMP TWITTER HANDLE HASHTAGS PHONE NUMBER MESSAGE TIMESTAMP
- 9. YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK ENTER SPLUNK
- 10. YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK WHAT THE HECK IS SPLUNK? ▸ Splunk consumes text and provides insights about the data contained within ▸ Splunk stores your historical data and allows you to look at how the baselines have changed over time ▸ Splunk helps identify anomalies which might affect business decisions ▸ Splunk allows people who know their data to share it with people who don’t
- 11. YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK WHAT THE HECK IS SPLUNK? REACTIVE PROACTIVE SEARCH AND INVESTIGATE PROACTIVE MONITORING AND ALERTING OPERATIONAL VISIBILITY REAL-TIME BUSINESS INSIGHTS
- 12. YOU’VE GOT JUNK IN YOUR SPLUNK - THE FUN NOW FOR THE FUN PART!
- 13. YOU’VE GOT JUNK IN YOUR SPLUNK - THE FUN CASE STUDIES AND EXAMPLES ▸ 7/11 - Uses Splunk to gain a business foothold in Indonesia, predicting shopping trends based on weather, among other things ▸ Information Security - Northern Arizona University uses splunk to trace intrusion attempts across our network ▸ Conner Swann (That’s Me) - Used splunk to glean metadata from text messages
- 14. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11) 7/11 - THE CLIMATE ▸ Expanding to a new market (2009) ▸ Had to offer an attractive alternative to existing businesses ▸ Offer local foods, became a place local teens would hang out ▸ Caused competitors to adapt to new climate, occupying new niches
- 15. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11) 7/11 - THE PROBLEM ▸ In order to retain their new customers, the company had to offer the best fast food as well as any daily necessities customers might need ▸ Necessitates a technological solution for providing behavioral insights on consumers ▸ Original data analytics solution was rigid, involved several rounds of manual analysis ▸ Analysis took 3-6 business days to complete ▸ Promotional campaigns took ~3 months to prepare
- 16. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11) 7/11 - THE SOLUTION ▸ 7/11 now uses Splunk for their POS analysis ▸ Assets are dynamically organized, delivering comprehensive overview of POS data from multiple perspectives ▸ System also leverages data from external systems (i.e weather, telecom) ▸ Data is processed in minutes instead of days
- 17. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11) 7/11 - THE RESULT ▸ Promotion planning time slashed by 80% - 2 weeks ▸ All people involved have access to the same data and visualizations with little training ▸ Promotions are evaluates for effectiveness as they occur ▸ ROI is apparent
- 18. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC) NAU INFORMATION SECURITY - EXAMPLE USE CASE ▸ Information Security is best when efforts are proactive ▸ Identify unwanted activity or actors and see if that data shows up anywhere else ▸ Honeypots on the network are used to collect data about intruders ▸ That data can be used to identify other anomalous behavior
- 19. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC) HOW IT WORKS Northern Arizona University Hacker IP Address: 68.55.90.112 Login Attempt From: 68.55.90.112 HoneyPot Louie Successful Login From: 68.55.90.112 Splunk Anomalous Events Detected: 68.55.90.112 Sources: - Honeypot - Peoplesoft
- 20. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC) THE IMPACT ▸ All event detection is done in real-time ▸ Incident response occurs as the event happen ▸ Remediation is simpler than in the past ▸ Easy to share impacts with non-technical people
- 21. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES
- 22. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES - THE WHY ▸ Personal analytics is HUGE ▸ Look for trends in communication ▸ Shows how much inferential data can be gleaned from behavior
- 23. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES - THE HOW ▸ Extracted messages from iPhone backup’s SQLite database
- 24. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES - THE RESULTS ▸ Average sentiment of outgoing texts over time ▸ index=text_messages is_from_me=1 | sentiment twitter message | timechart avg(sentiment) as sentiment span=1mon ▸ Conclusion: Sentiment fluctuates over time
- 25. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES - THE RESULTS ▸ Average sentiment of outgoing texts with baseline over time ▸ index=text_messages is_from_me=1 | sentiment twitter message |eval diff=sentiment-0.788400| eval count=count| timechart avg(diff) as sentiment, count span=14d ▸ Conclusion: Sentiment might correlate with life events and text message frequency
- 26. YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!) TEXT MESSAGES - THE RESULTS ▸ Comparing incoming sentiment with outgoing sentiment ▸ index=text_messages is_from_me=0 | sentiment twitter message | eval diff=sentiment-0.788400 | timechart avg(diff) as sentiment_from span=1mon | appendcols [search index=text_messages is_from_me=1 | sentiment twitter message | eval diff2=sentiment-0.788400 | timechart avg(diff2) as sentiment_me span=1mon] ▸ Conclusion: Outgoing sentiment is at times closely coupled with incoming sentiment
- 27. YOU’VE GOT JUNK IN YOUR SPLUNK - CONCLUSION PUT SOME JUNK IN YOUR SPLUNK! ▸ Splunk is free to play with ▸ (Developer Licenses are easy to come by) ▸ http://www.splunk.com/ ▸ Provide value to the shareholders!