SplunkLive! Customer Presentation – UMCP
4 likes1,047 views
This document discusses how Splunk has helped the University of Maryland improve security visibility and incident response. It provides an overview of the speaker and UMD, challenges they previously faced with scattered logs and limited visibility, and how Splunk has provided faster search capabilities and the ability to correlate data from multiple sources. Use cases described how Splunk has helped with real-world incident investigations, security alerts and threat response, breach detection, and compliance reporting. Best practices and lessons learned are also shared.
SplunkLive! Customer Presentation – UMCP
- 1. Copyright © 2015 Splunk Inc. Splunk @ The University of Maryland Kevin Shivers Manager, Security Technical Services
- 2. 2 Disclaimer This is not an endorsement of any company, product, or service by the University of Maryland or the State of Maryland. Any and all statements made in this presentation are mine and mine alone and do not in any way represent an official statement from the University of Maryland or the State of Maryland.
- 3. 3 Agenda • Introduction • About you, UMD, and myself • Our challenges and opportunities • Use Cases • Incident investigation • Breach detection • Compliance • Conclusion • Where we’re headed, best practices, and lessons learned
- 4. 4 About You New to Splunk? Seasoned vet? Security professional? IT Operations? Management? Auditor? Just here for the free t-shirt, food, and happy hour? Higher Ed? State of MD?
- 5. 5 About The University of Maryland, College Park • Founded in 1856 • Public research university • ~37,000 students, ~10,000 faculty/staff • Ranked 19th among public universities by U.S. News & World Report and 16th among public universities by Forbes • Extremely decentralized IT • Division of IT – Provides centralized IT services (network, e-mail, phones, mainframe, payroll, ”the ISP for campus”)
- 6. 6 About Me • Security practitioner for 17 years • At Maryland for 13 years • Stints at Sourcefire and Network Associates (McAfee) • Primary responsibilities • Lead the University’s IT security operations • Architecting new & better IT security technologies • Splunk user since April 2013 • Spent 4 days inside Chernobyl and lived to tell the tale
- 7. 7 About The Security Team • CISO • Security Engineering (1.5 FTE) – Develop and Implement new & better security technologies, integrations, project/product security assessments, data enrichment • Security Operations (2.5 FTE) – Incident Response & Investigation, forensics, eDiscovery, data enrichment, vulnerability assessment, data loss prevention • Compliance (3 FTE) – Conduct internal audits, coordinate external auditors • User awareness and education (1 FTE + 2 grad students) – IT misuse/harassment, DMCA complaints, education
- 8. 8 The Journey • Address manual log aggregation / searching • Perform correlation searching between multiple sources • Faster building of reports and dashboards, compliance requirement • Quick remediation of advanced persistent threat (APT) • Early breach detection, mitigation, improving security posture • Continuous monitoring 2013 2015 Security Use Cases Focus On Visibility
- 9. 9 Before Splunk: Scattered Logs, Limited Visibility • Challenges • Logs stored everywhere, hours lost logging in / searching • Networking, Unix, Windows, Email, etc. • Manual correlation required to find needle in haystack • Sec and Ops needed common searches, reports, visualizations • Legacy SIEMs lacked ad-hoc search capabilities • Pre-defined rules only, limited ability to dig through data • Response time measured in hours/days Needed a faster way to get answers
- 10. 10 With Splunk: Flexibility, Fast Time to Value • Solution • Aggregate multiple data sources • Not just security products but other IT assets / data sources • Build custom searches and reports into existing process • Ad-hoc searches, platform-independent UI • Continuous monitoring across entire infrastructure • Visibility needed for breach detection and IR • Response time measured in minutes “Splunk works how my group’s brain works”
- 11. 11 Real-time Campus Threat Detection/Prevention with Splunk Before Splunk Police discovered suspicious online posting at 3am Contacted us asking to track post to the sender Website with posting used: – Several distributed destination IP addresses – The source IP of one of our wireless NATs Manual effort to track the activity to an internal IP address and then a user took hours With Splunk Campus police again came in the wee hours w/ emergency need to trace a post back to user Again, posted via our wireless network with distributed web hosting presence With Splunk it took 2 minutes to trace & police were able to help the user
- 12. Use of Splunk at UMD Intrusion Detection Real-life Incident Investigations Security Alerts & Threat Response Compliance & Reporting “If Splunk were taken away from me, I would quit my job. It has solved a ton of problems and made my life so much easier.”
- 13. • Spike in a Windows usage log triggered alert • Ops team became suspicious while investigating that particular alert • Quick set of ad-hoc searches conducted • Clean handoff to sec team for forensic investigation / log review • APT eliminated from environment in days • During holiday season; most of that time spent on eradication and recovery vs detection and analysis • All handled in house vs having to bring in an external firm Use Case – Fast, Operationalized IR for APT
- 14. • APT type characterized based on incident • Phishing email origin, subsequent compromised accts / systems • Alerts configured based on APT detected • Auto-instantiated scripts to configure blocking rules • External pen test stymied couple months later • Proved effectiveness of remediation steps taken Use Case – Effective Remediation for APT
- 15. • Attacker able to compromise password reset tool • Able to change passwords – just those needed to access a critical database w/ 289k records • Within 30 minutes, team had contextualized entire breach • Chain of events from “users had not changed their passwords” to “database breached” • Focus on improving security posture, not investigating • Investigation took hours, not days or weeks • Avoided $250k- $500k cost for bring in external firm Use Case – Early Breach Detection
- 16. 16 Use Case – Compliance & Reporting Requirement to review logs for failed logins and reboots on network devices Splunk SDK for Python & Python-fu to search through logs and generate a ticket in our trouble ticketing system for SOC staff to review Auditable list of closed tickets to demonstrate logs are being reviewed and acted upon
- 17. 17 Dashboards
- 18. Splunk at University of Maryland • Data sources – Firewalls - IPS/NGFW – Wireless NAT logs - Linux server logs – Web server logs - Auth logs (AD/kerb/CAS/MFA) – Windows server logs - Mail server logs – Database audit logs - VPN logs • Data Volume • Indexing ~190 GB per day (doubled since 2013) • Waiting for more license capacity for another 200+ GB • Users 40 regular users 115 infrequent users 18 Universal Forwarders 4 Indexers General search head + ES Search Head
- 19. 19 Splunk Applications Splunk for Palo Alto Splunk App for AWS Cisco Security Suite Splunk on Splunk DB Connect Splunk SDK for Python
- 20. 20 Splunk Roadmap at UMD Single-site -> Multi-site Larger license & more logs Splunk as a Service for campus The great migration to AWS Metrics & dashboards for campus administrators Backend for self service tools for IT staff Splunk in the classroom More correlation & data enrichment More automation & integration
- 21. 21 Best Practices and Lessons Learned
- 22. 22 Best Practices and Lessons Learned Log the important things – Admin/privileged users logins (especially after hours) Regular user logins/failed logins useful too – New admin/privileged user accounts created – System/service restarts – Critical failures – Web access logs Interesting user agents, access to web app admin pages, SQLi, XSS attempts, etc – Access to DBs w/ PII – Virus/Malware alerts Review at least weekly, daily if you can (especially: after hours logins, new accounts, sys/service restarts, AV alerts)
- 23. 23 Best Practices and Lessons Learned GET THE FASTEST STORAGE YOU CAN AFFORD NTP is your friend – sync time early & often Define your searches as much as possible (NO index=*) Splunk’s documentation – Excellent resource. RTFM! Splunk Answers – great resource .conf – Like drinking from a fire hose Online user groups – Higher Ed mailing list hosted by OSU (contact me for info) 2
- 24. Thank You! Contact: kts@umd.edu Note: I am hiring – Security Engineer and SOC Analyst positions!