Solving Cyber at Scale
2 likes714 views
The document discusses the challenges of modern cybersecurity, highlighting the scale of threats from IoT devices and the cost of DDoS attacks. It emphasizes the need for real-time data processing and integration of machine learning to enhance threat detection and analyst efficiency. Apache Metron is presented as a scalable framework that improves cybersecurity through standardized data ingestion and community-driven development.
Solving Cyber at Scale
- 1. Solving Cyber at Scale with Hadoop, Storm and Metron
- 2. Simon Elliston Ball • Product Manager • Data Scientist • Elephant herder • @sireb
- 4. IoT: Mirai Reports of 1.2 Tbps 500,000 devices at peak DDoS attacks on Dyn DNS services
- 5. Insiders
- 7. Who are we up against?
- 9. Big Business • $tn market • Access is bought and sold: 5 bitcoin for 100m accounts • Sharing networks • Criminals as a Service • DDoS attacks: cost attackers $5 per hour, defenders ~$40k Sources: BT and KPMG Report, Taking the Offensive
- 10. Challenges for the Modern SOC
- 11. Drowning in Data
- 12. Staff shortage
- 14. What we have now
- 15. Silos Packet Store SIEM Log Store Forensics Tools Endpoint Agents Cases Threat Intel UEBA Anti Virus Email filter
- 17. Shiny new tools
- 18. Solutions: machine learning! magic! Triage Automation Detecting the unknown unknowns Explaining yourself
- 19. The value of real time Data in Motion: why wait until it’s at rest? Correct context: the world moved on
- 20. Better data = analyst efficiency Fully enriched data Real context Consistency = faster triage and better coverage
- 21. Single View of Business & Security Risks HR Finance Web Logs Security Appliances Email Syslogs Geolocation Network Data IoT Telemetry Data Operations CRM
- 22. Longer term data • Attacks last months • So should your queryable data
- 23. Executable solutions • Orchestration • Machine-time response
- 24. How to do it
- 26. Data Sources and Aggregation Open standards for data models = more productive data scientists + shareable models Business level data sources link security to real business risk.
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Services and Integration Layer Search and Dashboarding Portal Security Data Vault Community Analytical Models Provisioning, Management and Monitoring ModulesReal-time Processing Cyber Security Engine Telemetry Parsers Enrichment Threat Intel Alert Triage Indexers and Writers Cyber Security Stream Processing Pipeline Apache Metron: a framework for Big Data Driven cyber security Telemetry Ingest Buffer Telemetry Data Collectors Real-time Enrich / Threat Intel Streams Performance Network Ingest Probes / OtherMachine Generated Logs (AD, App / Web Server, firewall, VPN, etc.) Security Endpoint Devices (Fireye, Palo Alto, BlueCoat, etc.) Network Data (PCAP, Netflow, Bro, etc.) IDS (Suricata, Snort, etc.) Threat Intelligence Feeds (Soltra, OpenTaxi, third-party feeds) Telemetry Data Sources
- 30. Common data platform • Open Standards drive real use • Consistent SEMANTIC meaning, not just a type system
- 31. Community Development • http://metron.apache.org • https://github.com/apache/incubator-metron/
- 32. Thank you! • Apache Metron: http://metron.apache.org • Twitter: @sireb