Apache Metron Meetup May 4, 2016 - Big data cybersecurity
Download as PPTX, PDF10 likes3,856 views
The document provides an overview of Apache Metron, a security data analytics platform designed to address challenges in cybersecurity by offering real-time monitoring and analysis of network traffic. It describes Metron's architecture, use cases for data ingestion, enrichment, and integrating threat intelligence, as well as the capabilities to streamline security operations. The document also highlights the significant benefits of Metron in improving response times and reducing false positives for security analysts amidst a complex threat landscape.
Apache Metron Meetup May 4, 2016 - Big data cybersecurity
- 1. Apache Metron Meetup & Code Lab George Vetticaden Principal Architect @ Hortonworks Apache Metron Committer James Sirota Engineering Lead & Chief Data Scientist @ Hortonworks Apache Metron Committer
- 2. Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
- 3. Metron
- 4. Page4 The Good Guys Security Practitioner I have too many tools I need to learn I don’t have a centralized view of my data My tools are too expensive I can’t find enough talent I can’t keep relying on static rules I need to discover bad stuff quicker Most of my alerts are false positives I have too many manual tasks SOC Manager Threat landscape too dynamic More assets/users to manage Attack surface increases Legacy techniques don’t work anymore Metron will make it easier and faster to find the real issues I need to act on Metron is a more cost effective way for my team to deal with the fast moving threat landscape
- 5. Page5 The Bad Guys Advanced Persistent Threat Script Kiddie My techniques are predictable and known My attack vectors are also known You are not the only person I’ve attacked I brag about what I did or will do I set off a large number of alerts I fumble around a lot I am very unique in a way I do things I live on your network for about 300 days I know what I am after and I look for it, slowly Your rules will not detect me, I am too smart I impersonate a legitimate user, but I don’t act like one Metron can take everything that is known about me and check for it in real time Metron can model historical behavior of whoever I am impersonating and flag me as I try to deviate
- 6. Page6 Problems With Existing Tools Security Information Management System I am prohibitively expensive I have vendor lock-in I can’t deal with big data I am not open I am not extensible enough Legacy Point Tools I was built for 1995 I am super specialized I don’t scale horizontally I have a proprietary format You need a PhD to operate me Behavioral Analytics Tools I am mostly vapor ware I was built by a small startup I was modeled after a data set from 1999 I spam you with false positives
- 7. Page7 Apache Metron Vision “Apache Metron is a Security Data Analytics Platform (SDAP). As a next generation security analytics framework, it is designed to consume and monitor network traffic and machine data within an enterprise. Apache Metron is extensible and is designed to work at a massive scale. It is not a SIEM but rather the next evolution of a SIEM.” Apache Metron provides the following capabilities: Extensible spouts and parsers for attaching Apache Metron to monitor any telemetry source Extensible enrichment framework for any telemetry stream Hadoop-backed storage for telemetry stream with a customizable retention time Automated real-time index for telemetry streams enabling real-time search Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive ODBC/JDBC compatibility and integration with existing analytics tools
- 8. Challenges that Apache Metron Solves 60%: Percent of breaches that happened in minutes 8 months: Average time an advanced security breach goes unnoticed $400 million in estimated financial loss in 2015 70%-90%: Percentage of malware in breach unique to organization 2015 Verizon Data Breach Investigations Report • Too expensive to keep data for enough time to understand history • Not enough of the right data to provide context • Too expensive to collect all the desired data to understand context • Not sure if can detect a targeted event. • Too many events to review in timely manner • Not enough staff to review events in a timely manner
- 9. Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
- 10. Real-time Processing Engine PCAP NETFLOW DPI IDS AV EMAIL FIREWALL HOST LOGS PARSE NORMALIZE TAG VALIDATE PROCESS USER ASSET GEO WHOIS CONN ENRICH STIX Flat Files Aggregators Model As A Service Cloud Services LABEL PCAP Store ALERT PERSIST Alert Security Data Vault Apache Metron Logical Architecture Network Tap Custom Metron UI/Portals Real-Time Search Interactive Dashboards Data Modelling Integration Layer PCAP Replay Security Layer Data & Integration Services Apache Metron
- 11. Page11 Sensor A Sensor B Sensor N Topic A Topic B Topic (N) Apache Kafka PCAP PCAP Probe Physical Architecture Normalizing Topology A Normalizing Topology B Normalizing Topology N Apache Storm Native Format Native Format Native Format PCAP on HDFS Metron PCAP Service PCAP Topology Enrich Normalized Metron Format Enrichment/ Threat Intel Topology Out to Index + HDFS
- 12. Page12 Topic A Normalizing Topology A Sensor A Native Format Apache Kafka Apache Storm Kafka Spout Parser Kafka Bolt Enriched Metron JSON Parsing/Normalization Topology Key Points: • Each New Telemetry Data Source will have its own Parser Topology • Two types of Parsers available: Grok and Java
- 13. Page13 2 Types of Parsers Parser Type Description Telemetry Type Grok • A grok is a collection of named regular expressions. • Provides a declarative way to write new parsers without any code • A parser takes an input, which is usually a byte array coming from the Kafka Spout, and turns it into a Metron JSON Object. • The Grok parser does this by utilizing the Grok library inside of the Parser Kafka Bolt Adapter. • Use this parser when telemetry is simple to parse or low in volume Java • Java based approach to writing a custom parsers • Use this parser when telemetry is complex to parse or high volume
- 14. Page14 Metron JSON Object • Numerous sensors log in different formats. The parser should normalize at least the following subset of fields to the following Metron JSON naming conventions:
- 15. Page15 Enrich ment Bolt(a) Enrich ment Bolt(n) Threat Intel Joiner Message Splitter: Enrichment Enrich ment Joiner Message Splitter: Threat Intel Model Bolt (n) Threat Intel Bolt (n) Metron Enrichment Loader Framework Metron Threat Loader Framework Data Store Fast Cach e Fast Cach e Fast Cach e Fast Cach e Data Store Enrichment Topology Apache Kafka Enriched Writer Bolt = Message Stream Apache Storm = Enrichment Stream Enrichment Topology
- 16. Page16 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
- 17. Page17 Personas
- 18. Page18 Metron’s Key Functional Themes Platform Work done to harden the platform for performance, scale, extensibility and maintainability. This also includes capabilities around provisioning, managing and monitoring the application. Set of Data Sources that Metron provides capabilities to stream, ingest and parse into the platform. A set of Storm Topologies to perform various actions in real-time including: normalization of telemetry data, enrichment, cross reference with threat intel feeds, alerting, indexing, and persisting into Historical stores Data Collection Data Processing UI Set of portal, dashboard and user interfaces for the different personas.
- 19. Page19 Target Personas and Themes for Apache Metron 0.1 T e c h P r e v i e w 1 - I n t r o Theme: Platform Theme: Data Collection Theme: Data Processing Theme: UI Security Platform Engineer Security Platform Engineer Security Platform Engineer SOC Investigator Security Platform Engineer SOC Investigator Forensic Investigator SOC Investigator SOC Analyst SOC Manager
- 20. Page20 • Fully automated vagrant install of Metron on a single VM • Fully automated install of Metron on multi-node HDP cluster via Ansible scripts, Ambari blueprints and APIs including: • Multi-node Elastic Search Cluster • Metron-UI Web Application • Deployment of the Metron Storm Topology • Deployment of telemetry sensors: PCAP, Bro, YAF(Netflow), Snort • OpenSOC redesign (new topology structure, extensible enrichments, threat intel, data loads, configs, ease of adding new topologies) Platform Data Collection • Ingestion of the following data sources: PCAP via pycapa or C++ DPDK probe, Bro, Netflow via YAF, Snort • Parsers for the following data sources: PCAP, Bro, Netflow & Snort Data Processing • Support for the following enrichment services: Geo, WhoIs, Host • Threat Intelligence Message enrichment - Enrich messages with fields that mat the threat intelligence data in HBase • Support for the following persistence services: HDFS, HBase and Elastic Search • Indexing events and Alerts into Elastic Search cluster • Support for Soltra(CIF) Threat Aggregator Services via STIX and Taxii Feed • Ability to replay PCAP files for Testing UI • Metron Investigator UI to search across indexed events and alerts for SOC Analyst & Investigators • Histogram Panels for each of the data sources (YAF, Bro, Snort) • Table Views for Alerts (YAF, Bro, Snort) • Customize new panels with different data sources and different panel types. Key Features of Apache Metron 0.1
- 21. Page21 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
- 22. Page22 Why Metron? SOC Analyst Perspective Looking through alerts 25% Collecting contextual data 25% Formulating a Hypothesis 5% Investigate 20% Remediate 15% Update Workflow 5% Wrte Report 5% ANALYST WORKFLOW • Alerts Relevancy Engine • Smarter ML alerts • Centralized Alerts Console • Enriched with threat intel data • Fully enriched messages • Single pane of glass UI • Centralized real-time search • All logs in one place • Granular access to PCAP • Replay old PCAP against new signatures • Tag behavior for modelling by data scientists • Raw messages used as evidentiary store • Mine investigation history • Asset inventory as an enrichment • User identity as an enrichment • Workflow engine • Ticket clustering Everything you need to know in one place
- 23. Page23 Why Metron? Data Scientist Perspective Formulating a Hypothesis 5% Finding Data 20% Cleaning Data 20% Munging Data 20% Visualizing Data 20% Modelling Data 10% Validating Model 5% DATA SCIENCE WORKFLOW • All my data is in the same place • Data exposed through a variety of APIs • Standard Access Control Policies • Quickly see what I have • Metron normalizes objects • Partial schema validation on ingest • Tagging on ingest • Automatic data enrichment • Automatic application of class labels • Common Metron Objects • Massively parallel computation framework • Reusable Zeppelin Dashboards • Real-time search + UI • Integration with Python/R • Integration with analytics tools Reducing time from hypothesis to model
- 24. Page24 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
- 25. Page25 Use Case Setup • Scenario • Customer Foo has installed Metron TP1 and they are using the out of the box data sources (PCAP, YAF/Netflow, Snort and Bro). They love Metron! • But now they want to add new data source the the platform: squid proxy logs. • Customer Foo’s requirements are the following 1. Need to ingest the proxy events from Squid logs in real-time 2. The proxy logs has to be parsed into a standardized JSON structure that Metron can understand 3. In real-time, the squid proxy event needs to be enriched with domain/whois information (domain, cert, country, company) 4. In real-time, the domain of the proxy event must be checked against for threat intel feeds 5. If there is a threat intel hit, an alert needs to be raised 6. The end user must be able to see the new telemetry events and the alerts from the new data source
- 26. Page26 Squid & its Telemetry Event • What is Squid? • Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages • What does a Squid Access Log look like? • When you make an outbound http connection to https://www.cnn.com, the following entry gets added to a file called access.log: Unix Epoch Time IP of host where connection was made. The domain name of the outbound connection 1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
- 27. Page27 What Metron does to the Squid Telemetry Event in Real-time Convert from Unix Epoch to Timestamp Use Metron’s asset enrichment to enrich that IP (hostname, type of device) Use Metron’s WhoIs enrichment To look up domain name information (e.g: Use the Metron’s Threat Intel Services to cross-reference the IP with threat intel feed to see if there is a hit 1461576382.642 161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html Index the event into Elastic and persist into HDFS (Security Data Vault)
- 28. Page28 Real-time Processing Engine Squid Logs PARSE NORMALIZE TAG VALIDATE PROCESS USER ASSET GEO WHOIS CONN ENRICH STIX Flat Files Aggregators Model As A Service Cloud Services LABEL PCAP Store ALERT PERSIST Alert Security Data Vault Real-Time Search Interactive Dashboards Data Modelling Integration Layer PCAP Replay Security Layer Data & Integration Services Tracing the Squid Event across the Platform Custom Metron UI/Portals
- 29. Page29 Step 1: Telemetry Ingest (Tracing an Event) 1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
- 30. Page30 Step 2 – Process/Parse (Tracing an Event)
- 31. Page31 Step 3 – Enrich (Tracing an Event)
- 32. Page32 Enriching Data Architecture Metron Enrichment Store (HBase/) Enrichment Loader Framework Bulk Load Polling Enrichment Source Storm Bolt Cache Metron Streaming Messages Enriched Metron Streaming Messages
- 33. Page33 Step 4 – Label/Threat Intel (Tracing an Event) Threat Intel Store (HBase) Threat Intel Loader Framework Bulk Load Polling Storm Bolt Cache Metron Streaming Messages (Enriched) Enriched Metron Streaming Messages (Enriched) + Threat Intel Hits Threat Intel Feed Source (Optional) Threat Intel Aggregator
- 34. Page34 High level Steps – How to Add the New Telemetry 1. Create new Kafka topic for the new telemetry source called “squid” 2. Create and validate a grok statement file that parses the squid event log into a format that Metron can understand 3. Store that grok statement in HDFS 4. Create a new flux configuration for the new Squid parser Storm Topology. 5. Update Zookeeper with configuration to mark what fields in the telemetry to enrich and what fields to cross- reference with threat intel feeds. 6. Move the flux configuration to the host where you will deploy the topology. 7. Deploy the new squid Storm parser topology using the new flux configuration 8. Load WhoIs enrichment data and configure enrichment mapping 9. Load Threat Intel data and configure threat intel matching mapping 10. Use Apache Nifi to capture the squid events and push them into Metron 11. Create a new Panel in Kibana and see the telemetry events Key Points Easy Extensibility – The ability to add new data source without writing any code and in an easy mann Repeatable Pattern - The following represents a repeatable pattern that you can apply to most data s