SlideShare a Scribd company logo
ASK ME ANYTHING:
A CONVERSATIONAL INTERFACE TO AUGMENT
INFORMATION SECURITY WORKERS
Symposium On Usable Privacy and Security 2017
Bobby Filar, Rich Seymour, Matthew Park
BOBBY FILAR
 Data Scientist
 Background in (NLP)
@filar
 bfilar@endgame.com
RICH SEYMOUR
 Data Scientist
 Background in HPC
 @rseymour
 rseymour@endgame.com
2
WHO ARE WE?
MATTHEW PARK
 User Experience Designer
 Background in Big Data
 @muted_counts
 mpark@endgame.com
User-Centric Design
3
Three Stages
• Discovery
• Understanding our users,
confirming/disproving biases, capturing
organizational workflows
• Concepting
• Creating design requirements solutions
• Prototyping and User Testing
• Feature creation and taking it back into the
‘wild’ for testing
Our Initial Problems
4
Insufficient Resources
• Onboarding & training
new hires & Retention
• Limited time to review
alerts and incidents
Lack of easy-to-use
automated tools
• Difficult for non-
programmers to use
• Easy for programmers
to mess up!
Security platforms are just
difficult to use!
• Forces conformity
• Requires level of
expertise to extract value
User-Centric Design Study
 GOAL: Capture team dynamics and worker roles within security organization to
identify challenges common across security teams
5
User Group Team Type Environment Collection Method
A Traditional SOC Day-to-day use User interviews
B Novice Training Team Mock Scenario Side-by-side
monitoring,
Retrospective & User
interviews
C Internal Red vs. Blue Mock Scenario Mirrored Scenario as
User Group B
D Trad. SOC & Consulting group Day-to-day use User testing
Findings: Security Worker Roles
6
Tier 1
Analyst
Tier 3
Analyst
Forensic
Hunter
 Have little to no prior experience (average of 1 year)
in the cyber security space. First line of defense in a
Security Operations Center.
 Main responsibility is to initially triage alerts and
determine if escalation (to higher tiered) is required.
 Primarily rely on a platform’s GUI.
 Intimately understand network and platform architecture.
 Seen as domain experts on the SOC team and more
comfortable working through the command line.
 Investigates escalated alerts, and determine root causes
and extent to remediate problems.
 Expert in EDR platforms and sophisticated
investigation tools
 Uses command line and scripting languages to
bypass UI and collect large data feeds using 3rd
party APIs.
SOC
Manager
 Skilled security practitioners, not necessarily subject matter
experts.
 Extensive management experience, oversees day-to-day ops.
 Set schedules, assigns prioritization, generates reports.
Example: EDR Alert
Alert Type: Suspicious Binary
Alert Created: Feb 11, 2017
Severity: High
Confidence: 73%
File Path: C:Tempaaa.exe
File Size: 45700
MD5: 5d41402abc4b2a76b9719d911017c592
File Created: Feb 11, 2017
What do you do when there are 100s of these each day?
7
Example: EDR Alert
 Lacks context
• Is it actually bad?
• Is it anywhere else?
• Did it talk to the network?
 Lacks connectivity
• Is this alert tied to any others?
 Pivot on single IOC
• Hash
• Filename
• IP address
Alert Type: Suspicious Binary
Alert Created: Feb 11, 2017
Severity: High
Confidence: 73%
File Path: C:Tempaaa.exe
File Size: 45700
MD5: 5d41402abc4b2a76b9719d911017c592
File Created: Feb 11, 2017
What do you do when there are 100s of these each day?
8
Findings: Day in Life of a Security Analyst
Data
Deluge
Lack of
Context
Repetitive
Processes
Searching not
Analyzing
Lack of
Expertise
Lack of
Time
9
Design Requirements
10
1. Eliminate query syntax via natural language
2. Educate users on platform features
3. Provide context-driven alert triage
4. Recommend next steps
5. Expedite focused collection
Solution
 A Bot is an application that assists in
the automation of tasks
• Mimics human conversation
• Natural Language
Understanding determines user
intent
 Imagine an assistant that provides
ability to:
• Ask questions
• Execute workflows
• Educate users
• Recommend next steps
Hello, how can I help
you?
11
Natural Language vs Query Language
12
Query Language
SELECT * FROM TABLE process_event WHERE
process_name == “odinaff.exe”;
Natural Language
Search process event data for odinaff.exe
Reality
Is odinaff.exe on any endpoints?
Confidential and Proprietary 13
Interaction Types
Turn-based Conversation
User: Search processes
Artemis: Okay. Please provide a hash or filename
User: odinaff.exe
Artemis: Got it. Which endpoints would you like to target?
User: Windows 10 machines.
Artemis: Okay. Would you like to launch this search?
User: Yes
14
Interaction Types
Goal-oriented Conversation
User: Show me process event data for odinaff.exe on all Windows 10 endpoints
Artemis: Are you sure?
User: Yep!
15
Interaction Types
API-Driven Investigations
curl 'api/v1/event_search' -H "Content-Type: application/json" -H
'authorization: <api_key> --data-binary '{"intent":"search_process",
"parameters": {
"process_name":"odinaff.exe",
"filepath": "C:Temp*.exe"
}
}'
16
TRADITIONAL INVESTIGATION
1. Narrow scope to limited endpoints
2. Understand adversary TTPs
3. Gather events from limited endpoints
4. Analyze events from for signs of TTPs
5. Discover suspicious activity
6. Decode obfuscated commands
7. Pinpoint powershell activity
8. Expand scope to next set of endpoints
9. Repeat…
ENDGAME ARTEMIS
“Find powershell activity”
Powershell Misuse Investigation
Automatically discovers and
analyzes malicious activity across
your global enterprise in minutes
 17
Looking Forward
18
1. Collaboration
2. Chat Integration
3. Improve via Active Learning
Thank You
Contact:
bfilar@endgame.com @filar
rseymour@endgame.com @rseymour
mpark@endgame.com @muted_counts

More Related Content

PPTX
Data Visualizations in Cyber Security: Still Home of the WOPR?
PDF
Professional Hacking in 2011
PPTX
Security and Emotion: Sentiment Analysis of Security Discussions on GitHub
PPTX
Cognitive Analysis With SparkSecure
PDF
How to Use Artificial Intelligence to Minimize your Cybersecurity Attack Surface
PPTX
Cyber Threat Hunting: Identify and Hunt Down Intruders
PDF
STRIDE Variants and Security Requirements-based Threat Analysis (FFRI Monthly...
PPTX
Infiltrate 2015 - Data Driven Offense
Data Visualizations in Cyber Security: Still Home of the WOPR?
Professional Hacking in 2011
Security and Emotion: Sentiment Analysis of Security Discussions on GitHub
Cognitive Analysis With SparkSecure
How to Use Artificial Intelligence to Minimize your Cybersecurity Attack Surface
Cyber Threat Hunting: Identify and Hunt Down Intruders
STRIDE Variants and Security Requirements-based Threat Analysis (FFRI Monthly...
Infiltrate 2015 - Data Driven Offense

What's hot (14)

PDF
Fighting advanced malware using machine learning (English)
PDF
Enabling effective hunt teaming and incident response
PPTX
Strata 2015 Presentation -- Detecting Lateral Movement
PPTX
Threat Hunting 101: Intro to Threat Detection and Incident Response
PDF
Transforming Adversary Emulation Into a Data Analysis Question
PPT
Threat analysis-perception
PDF
Introduction of Threat Analysis Methods(FFRI Monthly Research 2016.9)
PPTX
Transforming incident Response to Intelligent Response using Graphs
PPTX
Crowd-Sourced Threat Intelligence
PDF
Resistance Isn't Futile: A Practical Approach to Threat Modeling
PPTX
Cybersecurity Incident Response Readiness: How to Find and Respond to Attacke...
PDF
Threat Hunting Report
ODP
Negative Unemployment and Great Job Satisfaction? Why infosec is AWESEOME
Fighting advanced malware using machine learning (English)
Enabling effective hunt teaming and incident response
Strata 2015 Presentation -- Detecting Lateral Movement
Threat Hunting 101: Intro to Threat Detection and Incident Response
Transforming Adversary Emulation Into a Data Analysis Question
Threat analysis-perception
Introduction of Threat Analysis Methods(FFRI Monthly Research 2016.9)
Transforming incident Response to Intelligent Response using Graphs
Crowd-Sourced Threat Intelligence
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Cybersecurity Incident Response Readiness: How to Find and Respond to Attacke...
Threat Hunting Report
Negative Unemployment and Great Job Satisfaction? Why infosec is AWESEOME
Ad

Similar to Ask me anything: A Conversational Interface to Augment Information Security workers (20)

PPTX
Filar seymour oreilly_bot_story_
PPTX
UX Workshop: How to design a product with great user experience
PDF
Software Analytics: Data Analytics for Software Engineering and Security
PPT
PHP & The secure development lifecycle
PDF
How to not fail at security data analytics (by CxOSidekick)
PDF
1435488539 221998
PDF
Software Analytics - Achievements and Challenges
PPTX
Why Pentesting is Vital to the Modern DoD Workforce
PPTX
Programming languages and techniques for today’s embedded andIoT world
PPT
01.intro
PPTX
Software Security Assurance for DevOps - Hewlett Packard Enterprise + Black Duck
PPTX
Software Security Assurance for DevOps
PPTX
Software Security Assurance for Devops
PPTX
How to improve Developer Documentations ?
PDF
Analyzing Big Data's Weakest Link (hint: it might be you)
PDF
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
PDF
Top 5 Machine Learning Tools for Software Development in 2024.pdf
PPTX
Software Analytics: Towards Software Mining that Matters (2014)
PPTX
When Platform Engineers meet SREs - The Birth of O11y-as-a-Service Superpowers
PPTX
Intro to INFOSEC
Filar seymour oreilly_bot_story_
UX Workshop: How to design a product with great user experience
Software Analytics: Data Analytics for Software Engineering and Security
PHP & The secure development lifecycle
How to not fail at security data analytics (by CxOSidekick)
1435488539 221998
Software Analytics - Achievements and Challenges
Why Pentesting is Vital to the Modern DoD Workforce
Programming languages and techniques for today’s embedded andIoT world
01.intro
Software Security Assurance for DevOps - Hewlett Packard Enterprise + Black Duck
Software Security Assurance for DevOps
Software Security Assurance for Devops
How to improve Developer Documentations ?
Analyzing Big Data's Weakest Link (hint: it might be you)
RSA 2015 Blending the Automated and the Manual: Making Application Vulnerabil...
Top 5 Machine Learning Tools for Software Development in 2024.pdf
Software Analytics: Towards Software Mining that Matters (2014)
When Platform Engineers meet SREs - The Birth of O11y-as-a-Service Superpowers
Intro to INFOSEC
Ad

Recently uploaded (20)

PPT
pump pump is a mechanism that is used to transfer a liquid from one place to ...
PPTX
Media And Information Literacy for Grade 12
DOCX
actividad 20% informatica microsoft project
PDF
Emailing DDDX-MBCaEiB.pdf DDD_Europe_2022_Intro_to_Context_Mapping_pdf-165590...
PPTX
building Planning Overview for step wise design.pptx
PPTX
Fundamental Principles of Visual Graphic Design.pptx
DOCX
The story of the first moon landing.docx
PDF
Trusted Executive Protection Services in Ontario — Discreet & Professional.pdf
PDF
BRANDBOOK-Presidential Award Scheme-Kenya-2023
PDF
YOW2022-BNE-MinimalViableArchitecture.pdf
PPTX
mahatma gandhi bus terminal in india Case Study.pptx
PDF
Interior Structure and Construction A1 NGYANQI
PPT
WHY_R12 Uaafafafpgradeaffafafafaffff.ppt
PDF
Integrated-2D-and-3D-Animation-Bridging-Dimensions-for-Impactful-Storytelling...
PPTX
AD Bungalow Case studies Sem 2.pptxvwewev
PPTX
YV PROFILE PROJECTS PROFILE PRES. DESIGN
PPTX
12. Community Pharmacy and How to organize it
PPTX
An introduction to AI in research and reference management
PDF
Skskkxiixijsjsnwkwkaksixindndndjdjdjsjjssk
PPTX
AC-Unit1.pptx CRYPTOGRAPHIC NNNNFOR ALL
pump pump is a mechanism that is used to transfer a liquid from one place to ...
Media And Information Literacy for Grade 12
actividad 20% informatica microsoft project
Emailing DDDX-MBCaEiB.pdf DDD_Europe_2022_Intro_to_Context_Mapping_pdf-165590...
building Planning Overview for step wise design.pptx
Fundamental Principles of Visual Graphic Design.pptx
The story of the first moon landing.docx
Trusted Executive Protection Services in Ontario — Discreet & Professional.pdf
BRANDBOOK-Presidential Award Scheme-Kenya-2023
YOW2022-BNE-MinimalViableArchitecture.pdf
mahatma gandhi bus terminal in india Case Study.pptx
Interior Structure and Construction A1 NGYANQI
WHY_R12 Uaafafafpgradeaffafafafaffff.ppt
Integrated-2D-and-3D-Animation-Bridging-Dimensions-for-Impactful-Storytelling...
AD Bungalow Case studies Sem 2.pptxvwewev
YV PROFILE PROJECTS PROFILE PRES. DESIGN
12. Community Pharmacy and How to organize it
An introduction to AI in research and reference management
Skskkxiixijsjsnwkwkaksixindndndjdjdjsjjssk
AC-Unit1.pptx CRYPTOGRAPHIC NNNNFOR ALL

Ask me anything: A Conversational Interface to Augment Information Security workers

  • 1. ASK ME ANYTHING: A CONVERSATIONAL INTERFACE TO AUGMENT INFORMATION SECURITY WORKERS Symposium On Usable Privacy and Security 2017 Bobby Filar, Rich Seymour, Matthew Park
  • 2. BOBBY FILAR  Data Scientist  Background in (NLP) @filar  bfilar@endgame.com RICH SEYMOUR  Data Scientist  Background in HPC  @rseymour  rseymour@endgame.com 2 WHO ARE WE? MATTHEW PARK  User Experience Designer  Background in Big Data  @muted_counts  mpark@endgame.com
  • 3. User-Centric Design 3 Three Stages • Discovery • Understanding our users, confirming/disproving biases, capturing organizational workflows • Concepting • Creating design requirements solutions • Prototyping and User Testing • Feature creation and taking it back into the ‘wild’ for testing
  • 4. Our Initial Problems 4 Insufficient Resources • Onboarding & training new hires & Retention • Limited time to review alerts and incidents Lack of easy-to-use automated tools • Difficult for non- programmers to use • Easy for programmers to mess up! Security platforms are just difficult to use! • Forces conformity • Requires level of expertise to extract value
  • 5. User-Centric Design Study  GOAL: Capture team dynamics and worker roles within security organization to identify challenges common across security teams 5 User Group Team Type Environment Collection Method A Traditional SOC Day-to-day use User interviews B Novice Training Team Mock Scenario Side-by-side monitoring, Retrospective & User interviews C Internal Red vs. Blue Mock Scenario Mirrored Scenario as User Group B D Trad. SOC & Consulting group Day-to-day use User testing
  • 6. Findings: Security Worker Roles 6 Tier 1 Analyst Tier 3 Analyst Forensic Hunter  Have little to no prior experience (average of 1 year) in the cyber security space. First line of defense in a Security Operations Center.  Main responsibility is to initially triage alerts and determine if escalation (to higher tiered) is required.  Primarily rely on a platform’s GUI.  Intimately understand network and platform architecture.  Seen as domain experts on the SOC team and more comfortable working through the command line.  Investigates escalated alerts, and determine root causes and extent to remediate problems.  Expert in EDR platforms and sophisticated investigation tools  Uses command line and scripting languages to bypass UI and collect large data feeds using 3rd party APIs. SOC Manager  Skilled security practitioners, not necessarily subject matter experts.  Extensive management experience, oversees day-to-day ops.  Set schedules, assigns prioritization, generates reports.
  • 7. Example: EDR Alert Alert Type: Suspicious Binary Alert Created: Feb 11, 2017 Severity: High Confidence: 73% File Path: C:Tempaaa.exe File Size: 45700 MD5: 5d41402abc4b2a76b9719d911017c592 File Created: Feb 11, 2017 What do you do when there are 100s of these each day? 7
  • 8. Example: EDR Alert  Lacks context • Is it actually bad? • Is it anywhere else? • Did it talk to the network?  Lacks connectivity • Is this alert tied to any others?  Pivot on single IOC • Hash • Filename • IP address Alert Type: Suspicious Binary Alert Created: Feb 11, 2017 Severity: High Confidence: 73% File Path: C:Tempaaa.exe File Size: 45700 MD5: 5d41402abc4b2a76b9719d911017c592 File Created: Feb 11, 2017 What do you do when there are 100s of these each day? 8
  • 9. Findings: Day in Life of a Security Analyst Data Deluge Lack of Context Repetitive Processes Searching not Analyzing Lack of Expertise Lack of Time 9
  • 10. Design Requirements 10 1. Eliminate query syntax via natural language 2. Educate users on platform features 3. Provide context-driven alert triage 4. Recommend next steps 5. Expedite focused collection
  • 11. Solution  A Bot is an application that assists in the automation of tasks • Mimics human conversation • Natural Language Understanding determines user intent  Imagine an assistant that provides ability to: • Ask questions • Execute workflows • Educate users • Recommend next steps Hello, how can I help you? 11
  • 12. Natural Language vs Query Language 12 Query Language SELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”; Natural Language Search process event data for odinaff.exe Reality Is odinaff.exe on any endpoints?
  • 14. Interaction Types Turn-based Conversation User: Search processes Artemis: Okay. Please provide a hash or filename User: odinaff.exe Artemis: Got it. Which endpoints would you like to target? User: Windows 10 machines. Artemis: Okay. Would you like to launch this search? User: Yes 14
  • 15. Interaction Types Goal-oriented Conversation User: Show me process event data for odinaff.exe on all Windows 10 endpoints Artemis: Are you sure? User: Yep! 15
  • 16. Interaction Types API-Driven Investigations curl 'api/v1/event_search' -H "Content-Type: application/json" -H 'authorization: <api_key> --data-binary '{"intent":"search_process", "parameters": { "process_name":"odinaff.exe", "filepath": "C:Temp*.exe" } }' 16
  • 17. TRADITIONAL INVESTIGATION 1. Narrow scope to limited endpoints 2. Understand adversary TTPs 3. Gather events from limited endpoints 4. Analyze events from for signs of TTPs 5. Discover suspicious activity 6. Decode obfuscated commands 7. Pinpoint powershell activity 8. Expand scope to next set of endpoints 9. Repeat… ENDGAME ARTEMIS “Find powershell activity” Powershell Misuse Investigation Automatically discovers and analyzes malicious activity across your global enterprise in minutes  17
  • 18. Looking Forward 18 1. Collaboration 2. Chat Integration 3. Improve via Active Learning
  • 19. Thank You Contact: bfilar@endgame.com @filar rseymour@endgame.com @rseymour mpark@endgame.com @muted_counts

Editor's Notes

  • #2: Summary Security operations center (SOC) teams are burdened with a deluge of alerts, repetitive processes for data analysis, and lack of skills and tools to stop advanced threats. To address these challenges, it is crucial to empower junior analysts to stop advanced threats before damage and loss occurs.  Just as digital assistants like Siri or Alexa have proved their ability to give time back to our day by tackling tasks, a security chatbot can streamline workflows, perform complex tasks, and make recommendations to the SOC analyst. Using a combination of subject matter expertise from SOC analysts and the power of machine learning, chatbots can help teams overcome resource shortcomings by using conversations to offload data collection and guide analysts through recommended courses of action. This process provides an intuitive interface to remediation/investigation workflows and complex storage structures so the analyst can spend less time on collection efforts and more on analysis and response.
  • #4: So lets address the security industry as a whole. Generally, there are streams of new exploits types and malicious attacks constantly threaten enterprises environments To combat this, security products have been created to detect and respond and remediate problem areas - in their environment But for these products to be deployed successfully, the right teams had to be employed to monitor and act on these suspicious and malicious events. These teams are know as SOC teams – Security Operations Center – Who physically sit in these centers and remediate a stream of alerts. When approaching our predisposed soc problem (which we will discussed in the next slide) – We wanted to better understand our soc users- In order to create the best tools for own personas, we wanted to understand who they were and their daily workflow habits.  We split this approach into three distinct phases: 1) Discovery; 2) Concepting and 3) Prototyping and User Testing
  • #5: Bascially… what the bosses said... Forces conformity as opposed to integrating into teams & workflows
  • #6: I can get into this more in the Q/A - but during this discovery phase.. confirming our biases/ capturing team dynamics and creating user personas.
  • #7: When a SOC analyst starts their shift, they first participate in the shift handover from the analysts on the previous shift. Here they get a briefing on current ongoing investigations or open alerts, ticket numbers associated with those alerts, whois assigned to the investigations, and anything that needs attention. They then monitor a SIEM, an assigned endpoint UI dashboard, email and wait for a security event to occur. Typically that doesn't take very long - with the amount of tools generating alerts in a typical SOC environment (especially those monitoring large networks) - getting alerts is not the problem. Determining which alerts to focus on is the problem. The analyst is typically in reactive mode - where they respond to alerts as they come in, quickly identifying the high priority alerts. Typically, Tier 1 Analysts will have little to no authority to take immediate action on suspected malicious security events and will instead move the alert up the SOC chain. When escalating they will create a case/investigation/incident and assign that case/investigation/incident to the SOC investigator (or Tier 3 analysis). Both the SOC Investigator and the Tier 3 Analyst will take further steps in verifying the anomalous event, and will take the proper response in remediating the alert. While the Tier 3 Analyst will also sift through a SIEM alongside the Tier 1 Analyst, SOC Investigators will often times only work on escalated alerts. At the end of the shift, all levels of analysts needs to prepare a report of the alerts triaged, what was resolved, and what is still open in order to handover the activities to the oncoming shift. Larger reports depicting alert and investigation trends are generated for a SOC Manager on a daily or weekly basis. The SOC Manager will use these reports to focus in on key metadata in the coming weeks, determine the SOC shift schedule and build a custom summary report for the organization's executive level.
  • #10: Analysts must battle a tide of incoming alerts Data deluge Morning coffee situation Too much data, too many sources AV EDR Firewall Application logs Threat Intel Feeds No Context No shared context Hard to tie the events from feed A to the events from feed B Repetitive Searching through event logs can be a repetitive process. Alert, Search, Find or Not, Repeat Can we automate this work without creating another data stream? Searching not analyzing Lack of expertise Even tier 3 analysts have blind spots Tier 1 analysts want to build skills and be effective as the first line of defense Can a security product teach new users how to use it effectively while supporting users all skill levels? Lack of time
  • #11: Analysts must battle a tide of incoming alerts Data deluge Morning coffee situation Too much data, too many sources AV EDR Firewall Application logs Threat Intel Feeds No Context No shared context Hard to tie the events from feed A to the events from feed B Repetitive Searching through event logs can be a repetitive process. Alert, Search, Find or Not, Repeat Can we automate this work without creating another data stream? Searching not analyzing Lack of expertise Even tier 3 analysts have blind spots Tier 1 analysts want to build skills and be effective as the first line of defense Can a security product teach new users how to use it effectively while supporting users all skill levels? Lack of time
  • #13: User: Show me process event data for odinaff.exe on all Windows 10 endpoints Artemis: Okay.
  • #15: User: Show me process event data for odinaff.exe on all Windows 10 endpoints Artemis: Okay.
  • #17: User: Show me process event data for odinaff.exe on all Windows 10 endpoints Artemis: Okay.
  • #19: Analysts must battle a tide of incoming alerts Data deluge Morning coffee situation Too much data, too many sources AV EDR Firewall Application logs Threat Intel Feeds No Context No shared context Hard to tie the events from feed A to the events from feed B Repetitive Searching through event logs can be a repetitive process. Alert, Search, Find or Not, Repeat Can we automate this work without creating another data stream? Searching not analyzing Lack of expertise Even tier 3 analysts have blind spots Tier 1 analysts want to build skills and be effective as the first line of defense Can a security product teach new users how to use it effectively while supporting users all skill levels? Lack of time