End-to-End Security Analytics with the Elastic Stack
4 likes3,581 views
The document discusses the challenges of end-to-end security analytics using the Elastic Stack, emphasizing the importance of scalability and real-time processing in handling exploding security data and new threats. It outlines a cybersecurity maturity model consisting of three phases: security event management, automation for faster detection and response, and proactive analytics through behavior profiling and machine learning. Finally, it highlights the need for effective threat hunting and intelligence to reduce cognitive load and improve detection speed.
End-to-End Security Analytics with the Elastic Stack
- 1. End-to-End Security Analytics with the Elastic Stack Matteo Rebeschini Solutions Architect - Security Specialist Elastic{ON} Denver October 30, 2018
- 5. Scalability is more than a checkbox Impact • Speed suffers at scale • Restricts ad hoc exploration Security Data Exploding # 1
- 6. Scalability is more than a checkbox New threats every day # 2 Impact • SIEM rules fall short • Time to detection is crucial
- 7. Scalability is more than a checkbox Volume pricing not viable # 3 Impact • Forced to throw away data • Limits security coverage
- 9. Scalability is more than a checkbox Elastic Edge • Scalable from start • Distributed by design • Real time at scale Security Data Exploding # 1
- 10. Scalability is more than a checkbox New threats every day # 2 Elastic Edge • Everything is indexed • Snappy search at scale • Do more with machine learning
- 11. Scalability is more than a checkbox Volume pricing not viable # 3 Elastic Edge • Not volume-based pricing • Store & search everything • Limited only by your creativity
- 12. Cybersecurity Maturity Model Phase I Security Event Management Phase II Automation Phase III Proactive Analytics
- 13. Security Event Management Foundation for Effective Security Analysis Phase I Security Event Management
- 14. • Collect all parts of the puzzle • Normalize for aggregation and correlation across sources • Enrich to extend attributes available for analysis • Index data for fast search and analytics Security Event Management Collect Normalize Enrich Index
- 15. Collection from All Data Sources Domain Data Sources Timing Tools Network PCAP, Bro, NetFlow Real time, Packet-based Packetbeat Logstash (netflow module) Application Logs Real-time, Event-based Filebeat Logstash Cloud Logs, API Real-time, Event-based Beats Logstash Host System State, Signature Alert Real-time, Asynchronous Auditbeat Filebeat (Osquery module) Winlogbeat Active Scanning User-driven, Asynchronous Vulnerability Scanners
- 16. Normalization ● Defines a common set of fields for ingesting data into Elasticsearch. ● Helps you correlate data from different log source types ● Designed to be extensible ● Details and community feedback @ https://github.com/elastic/ecs Using Elastic Common Schema (ECS)
- 17. Normalization and Enrichment Kibana Beats Logstash Elasticsearch Datastore Web APIs Ingest Nodes (X) Using Beats and/or Logstash
- 18. FileBeat Log Files MetricBeat Metrics PacketBeat Network Data WinLogBeat Window Events Plus, more than 30 community Beats and growing… HeartBeat Uptime Monitoring AuditBeat Audit Data
- 19. Logstash Inputs Beats … … JDBC … … TCP UDP HTTP Filters Extract Fields Geo Enrich Lookup Enrich DNS Lookups Pattern Matching ArcSight Codec …Network / Security Data Syslog Servers Infra / App Data IoT / Sensors Persistent Disk Based Queues Normalization and Enrichment Beats Outputs Elasticsearch … … … … … Kafka RabbitMQ RDBMS Centralized Configuration Management Elasticsearch Using Logstash
- 20. Automation For Faster Detection and Response Phase II Automation
- 21. Alerting on Threats The Mechanics of a Watch
- 22. Detecting anomalies with threshold-based alerts is hard! • Defining “normal” via static thresholds is hard • Rules don’t evolve with data / infrastructure • Rules can be bypassed What’s the right threshold?
- 23. Understand Seasonality Reduce False Positives Avoid Manual Threshold Revision The advantages of anomaly-driven alerting Identify Areas of Focus Using Machine Learning
- 24. Integrating Alerts with Other Systems Robotic Process Automation Security Applications SDN Switches Messaging Services Issue Tracking Services Elasticsearch
- 25. Proactive Analytics Be the Hunter Phase III Proactive Analytics
- 26. Behavior Analytics Entity Profiling with Machine Learning When something behaves like itself Monday Tuesday Wednesday Thursday When something behaves like its peers
- 27. Host Behavior • Free disk space lower than average • Unusual log entries Network Behavior • Unusual connections between hosts • Higher than average data transfer Application Behavior • Service response time abnormally high • Dropped connections exceed normal When Behavior Matters User Behavior • Unusual authentication activity • Unusual file access
- 28. What is Threat Hunting? Surveillance & Reconnaissance oriented on the adversary which is informed by Intelligence.
- 29. Threat Modeling Who is your adversary? What is their motivation? What is the impact of a successful attack? What are they targeting? Developing and testing hypothesis
- 30. What are you looking for? Hypothesis Investigation New Patterns and IOA IOCs Inform and Enrich Different data sets Identify the patterns Feed the IOCs back create new alerts to improve the speed of the detection Operations IntelligenceIntelligence The Intelligence Feedback Loop
- 31. Speed is imperative… Reduce Cognitive Load
- 32. The Elastic Stack Enables Quick Iteration Speed is King Eliminate blind spots by using all your data Investigate threats more quickly and efficiently Reduce dwell time by identifying threats earlier
- 33. Cybersecurity Maturity Model Phase I Security Event Management Phase II Automation Phase III Proactive Analytics
- 34. www.elastic.co