Near Real-Time Outlier Detection and Interpretation
Download as PPTX, PDF1 like1,096 views
The document discusses AT&T's use of a Hadoop-based approach for near real-time outlier detection and interpretation of cyber threats. It provides context on the challenges of detecting threats at AT&T's network scale and data volume. The presentation outlines AT&T's history of threat capabilities, the need for a big data solution, and their transition to a Hadoop-based threat analytics platform for ingesting and analyzing over 5 billion daily network events to detect outliers and threats.
Near Real-Time Outlier Detection and Interpretation
- 1. © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. June 28, 2016 Near Real-time Outlier Detection and Interpretation An Hadoop Based Approach Hadoop Summit 2016 Bob Thorman Principal – Technology Security AT&T Chief Security Organization
- 2. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 2 Presentation Outline: Brief Context of the Problem of Cyber Threats in our industry Recent History of AT&T Cyber Threat Capabilities Hadoop Based Approach to Threat Analytics Platform Cyber Threat Detection and Interpretation Insider Threat
- 3. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. The Problem of Cyber Threats in Our Industry A Brief Context
- 4. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 4 Network Scale • ~1M Authenticated users • ~800K user oriented devices • ~1100 security devices on the network (FW, IDS, etc.) • Approximately 5B network events per day – Firewall, Proxy, IDS, SIEM, etc. Facing Alarming Trends Bridging to the Internet • Next Slides The Problem of Cyber Threats in Our Industry
- 5. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. 5 Distributed Reflection DoS (DrDoS) Attack Evolution Attack activity trending up Oct 2013 1900/udp: SSDP 123/udp: NTP 19/udp: chargen 0/udp: packet fragmentation 53/udp: DNS (some legitimate)30 months shown
- 6. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Recent History of AT&T Cyber Threat Protection Capabilities A Need for Big Data
- 7. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. History of AT&T Cyber Threat Protection Capabilities Chief Security Office – 2002 Program concept for millions of records per day – 2005 Program concept tens of millions of records per day – 2016 Big Data concept for tens of billions events/day – 2017 Big Data concepts for trillions events/day Major Big Data Development Milestone – 2008 Beginnings of Accumulo, an implementation of Google™ Bigtable – 2011 Accumulo open sourced to Apache Software Foundation – 2013 AT&T initiates Threat Analytics modernization project – 2014 AT&T initiates deployment of Hadoop-based Threat Analytics Platform Cyber Threat Protection Platform Architecture Evolution – Next slides 7
- 8. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Threat Platform of Yesterday SIEM 8 Source/processing/analytics DBMS/SAN Query
- 9. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Threat Detection and Interpretation Process 9 Architectural Component Ingestion Outlier Detection1 Spark Streaming Detectors1 R Analytics1 Web UI Dashboards Custom Alerting Framework1 Threat Operations 1Area of focus for automation
- 10. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. An Hadoop Based Approach to Threat Analytics Platform Securing AT&T with Hadoop
- 11. Detecting and Interpreting Cyber Threats at AT&T © 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners. AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement. Today’s Platform Details Using An Hadoop Based Platform for Log Management, Threat Analysis, Reporting AT&T approach to use of Hadoop in a Threat Analysis Platform SIEM Raw logs Events, Intelligence, Alarms, Threats Results, Reports, Analytics Source Processing Threat Analytics Platform UI/Visual/Report 11
Editor's Notes
- #2: Introduce Adam Introduce myself
- #3: Work real quick through agenda Just set the stage for an Hadoop based threat analytics platform that has NRT capabilities
- #5: Set the stage for how a typical network in this industry and how much work there is for securing it. Presents an industry problem, not an AT&T problem Address the outside threat to the internal operation of our industry
- #6: Amount of traffic related to reflect based DoS attackers. Illustrates activity on the internet not the attacks against the AT&T perimeter. Hack-ma-geddon Columbia government Spam Hause Syria <- New York Times Target lost 40M credit/debit cards
- #8: Our TAP has evolved a lot over the last few year as we’ve moved into an Hadoop base architecture. I will briefly describe the roadmap. Proprietary technology and lack of extensibility are killers
- #9: Past was SIEM dependent, based on large RDBMS and exclusively dependent on human detection and interpretation. Largely a data reduction system. Industry solution of yesterday.
- #10: The challenge is the cognitive intersection with automation. An environment of innovation. Goal is to automate the security analysis process which are largely cognitive. Granted this is a different use of Hadoop rather than single use data. Its continual ingestion, NRT detections, alerting, etc. Not always a clear problem statement. Spend some time developing the human dependency and cognitive processes Takes a lot of data
- #12: Left to right, we move all the data through various processing platforms into an Hadoop base system for raw log management, data org, management, access, analysis and finally to visualization and reporting.