![Ethnography of the SOC
“The profession [security] is so
nascent that the how-tos have
not been fully realized even by
the people who have the
knowledge…the process
required to connect the dots is
unclear even to analysts.
Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to
studying CSIRTs. Network, 100, 2.](https://image.slidesharecdn.com/artintoscience-2017-investigationtheory-170125164216/85/Art-into-Science-2017-Investigation-Theory-A-Cognitive-Approach-5-320.jpg)
Art into Science 2017 - Investigation Theory: A Cognitive Approach
- 2. Chris Sanders (@chrissanders88) Analyst @ FireEye Founder @ Rural Tech Fund PhD Researcher GSE # 64 BBQ Pit Master Author: Practical Packet Analysis Applied NSM Investigation Theory Course
- 3. Symptoms of a Cognitive Crisis 1. Demand for expertise greatly outweights supply 2. Most information cannot be trusted or validated 3. Inability to mobilize and tackle big systemic issues
- 4. Ethnography of the SOC “An analyst’s job is highly dynamic and requires dealing with constantly evolving threats. Doing the job is more art than science. Ad hoc, on-the-job training for new analysts is the norm." Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.
- 5. Ethnography of the SOC “The profession [security] is so nascent that the how-tos have not been fully realized even by the people who have the knowledge…the process required to connect the dots is unclear even to analysts. Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.
- 6. Symptoms of a Cognitive Crisis 1. Demand for expertise greatly outweights supply 2. Most information cannot be trusted or validated 3. Inability to mobilize and tackle big systemic issues
- 7. The Cognitive Revolution 1. Understand the processes used to draw conclusions 2. Develop repeatable methods and techniques 3. Build and advocate training that teaches practitioners how to think
- 9. Mapping the Investigation Sample: Novice and expert analysts Methodology: 30+ case studies Stimulated recall interviews Focus on individual investigations of varying types Perform key phrase analysis – analyze results
- 10. Key Phrase Mapping Dual Process Theory Intuition: Implicit, unconscious, fast Reflection: Explicit, controlled, slow Intuition Experimentation Restructuring Imagination Incubation Metacognition Evaluation Goal Setting Making Plans Reflection Analytically Viewing Data Rule-Based Reasoning Considering Alternatives
- 11. Results Novices Experts Intuition Metacognition Reflection
- 12. Analyzing the Flow of the Investigation
- 13. Investigations as Mental Labyrinths The investigation is the core construct of information security. How do we study them when everyone has a different toolset? Follow the Data! Alert OSINT Reputation File Hash Sandbox Behaviors AV Detections (VT) Imphash More File Hashes Friendly Host Network PCAP Host Windows Logs Security Log System Log App LogRegistry File System Hostile Host Network PCAP Flow
- 16. What data did analysts look at first? 72% 16% 12% Observed PCAP Flow OSINT Data Suggests: Analysts prefer a higher context data set… …even if other data sets are available …even if lower context data sets can lead to a resolution.
- 17. Did the first move affect analysis speed? Data Suggests: While PCAP provides richer context, it may slow down the investigation if that’s where you start Starting with a lower context data source can increase speed when working with higher context data 16 10 9 PCAP Flow OSINT Avg Time to Close
- 18. What happens when Bro data replaces PCAP? 46% 25% 29% Observed (Bro) Bro Flow OSINT 72% 16% 12% Observed (PCAP) PCAP Flow OSINT
- 19. What happens when Bro data replaces PCAP? 16 10 9 PCAP Flow OSINT Avg Time to Close (PCAP) 10 10 11 Bro Flow OSINT Avg Time to Close (Bro) Data Suggests: Better organization of high context data sources can yield improvements in analysts performance
- 20. What data sources were viewed most and least frequently? Data Suggests: Network data is used more frequently than host data… …even when host data can be used exclusively to resolve. …even when easy access is provided to host sources. Revisting data is more prevalent on higher context data sources Data Sources Viewed Data Sources Revisited PCA P 84% Flow 11% OSIN T 5%
- 21. How many steps were taken to make a disposition judgement? Data Suggests: At some point, the number of data sources you investigate impacts the speed of the investigation Understanding where data exists and when to use it can impact analysis speed 6 12 9 3 0 5 10 15 6-10 11-15 16-20 21-25 Number of Steps 9 12 14 24 0 5 10 15 20 25 30 6-10 11-15 16-20 21-25 Avg Time to Close
- 22. Did analysts investigate friendly or hostile systems first? 9% 91% Observed Friendly Hostile Data Suggests: Analysts are more compelled to investigate unknown external threats than internal systems Analysts don’t fully understand their own techniques 41% 59% Friendly Friendly Hostile
- 23. Thank You! Mail: chris@chrissanders.org Twitter: @chrissanders88 Blog: chrissanders.org Training: chrissanders.org/training
Editor's Notes
- #4: Every town had one doctor and they were also your vet Many home remedies spawn from this time – milk as a treatment for stomach ulcers is an example Major health crises were frequent and impossible to control
- #5: Anthroplogists Ethnography
- #6: Is this an individual thing, or is it a systemic problem?
- #7: Every town had one doctor and they were also your vet Many home remedies spawn from this time – milk as a treatment for stomach ulcers is an example Major health crises were frequent and impossible to control
- #15: We ended up with an investigation game
- #17: Sidebar: Analysts looked at the PCAP 100% of the time, even if it wasn’t necessary.
- #21: This points to tendencies gained from training. Most shops don’t have easy access to host data.
- #22: Anecdotal – Experts I knew took less than 10 steps. Anecdotal – Novices I knew took > 15.