Putting the Human Back in the Loop for Analysis
- 1. Putting the Human Back in the Loop for Analysis Andy Piazza | @klrgrz | /in/andypiazza/
- 2. whoami ▪ Facility Security Officer for phia LLC ▪ ThreatAnalyst with experience in counter- terrorism and counter-narcotics operations ▪ Built ground-up & overhaul processes for multiple orgs ▪ Passionate about information sharing, team building, and problem solving @klrgrz https://www.linkedin.com/in/andypiazza/ #doorkickingtokeyboardclicking
- 3. That’s who I am- and I have an AGENDA! ▪ Understanding the Reality ofYour Environment ▪ TheValue of “SystemThinking” and Understanding Information Flow ▪ A CyberThreat Intelligence (CTI) Framework ▪ The Power of Collaboration for CCAA ▪ A Few KeyTakeaways ▪ Resources, References, and Hat/Tips
- 4. As Seen on TV TheValue of “SystemThinking” and Information Flow
- 5. The Vendor View to Defense in Depth
- 6. Realworld View of Defense in Depth ▪ Well… that depends on where you work ▪ This means we have real work to do too… can’t rely on theT800 yet
- 7. That was funny, so what? ▪ Know what your environment REALLY looks like and that every environment is different – Inventory your hosts, IP space, applications- DEFINE “NORMAL” – Don’t allow mental models bias your analysis ▪ Know what your tools are REALLY capable of – What tools do you have that can integrate with eachother? – Learn more skills than what was covered in the demo! ▪ Understand where and how you fit into business operations – AKA the real reason we have jobs anyway
- 8. A picture is worth… TheValue of “SystemThinking” and Understanding Information Flow
- 9. Technical vs. Strategic Level Analysis Initial Trigger Logs Trigger Event Collect artifacts Get coffee Insider Threat? Is it APT? Malware? What’s in open source? whois? pDNS? DNS? Lateral movement? Work magic using secret-sauce Get more coffee & take second look Submit final report to leadership Close Ticket
- 10. One Time in Haiti, I learned about flow
- 11. What I actually learned ▪ Systems, seating, and personnel assignments MUST be positioned around the flow of information – You can’t achieve this without understanding the full path information flows through your environment ▪ Technical AND strategic levels of insight are required to develop flow – Too much “planning” is done by planners instead of analysts – If you’re invited to the meeting, GO and be the voice of the analyst ▪ Can you ELI5?
- 12. Displaying Flow Effectively Trigger Event Collect artifacts Get coffee Insider Threat? Is it APT? Malware? What’s in open source? whois? pDNS? DNS? Lateral movement? Work magic using secret-sauce Get more coffee & take second look Submit final report to leadership Close Ticket Trigger Event Analysis Mitigation Final Report CloseTicket Trigger Event Tier 1 Analysis Tier 2 Escalation Deploy Signatures Final Report CloseTicket ▪ Visio diagrams normally suck ▪ Used correctly, they can speak to multiple levels of leadership – Helps “strategic” thinkers see the details faster with minimal words – Visualize decision impact – Possibly even create realistic KPIs
- 13. Where we normally go wrong ▪ Not understanding the bigger picture – Leads to misaligned efforts and wasted resources – Makes team members feel disconnected and undervalued ▪ Not participating in business mapping meetings because “it’s clearly a waste of time and analysis is an art” – You can have your art and map it too ▪ Basic point- our role in security is to support & enable operations – One time in Iraq, an operations sergeant forgot his place in the machine
- 14. …wait for it… A CyberThreat Intelligence (CTI) Framework
- 15. Data Disclaimer ▪ The IOCs are all from the open web – Markings in the following screenshots are demo purposes only ▪ The malware names, CVEs, and related correlation is all made up too… ▪ The IOCs are actual “badness” so please don’t Google yourself into a problem
- 16. Collect, Catalog, Assess, Act (CCAA) ▪ Original construct was a theory I came up with for CTI – Processing reports for actor, malware family, CVE, etc. – Processing IOCs for kill chain, sharing restrictions, STIX type, etc. ▪ Can be applied to any dataset – Contact lists => CRM – Threat feeds => threat analysis – Financials => budget and forecasting ▪ CCAA is about the process and possibilities Collect Catalog Assess Act
- 17. Collect Phase ▪ A lot of tools exist – Threat Intelligence Platforms automate a lot of this for IOCs – SIEMs do this for logs, vuln management data, etc. ▪ Let’s be honest, we do a lot this manually still ▪ The key to collection is standardization in formats
- 18. Collection Examples ▪ Threat Intelligence Platforms, open-source feeds, and paid services ▪ Brain dumps in Notepad during IR ▪ Those email tippers we don’t talk about
- 19. Common Collection Samples FAKE CORRELATOIN & MARKINGS FOR DEMO PURPOSES ONLY
- 20. Catalog Phase ▪ Here’s where the real work begins and the bulk of it gets done ▪ Cataloging requires identifying a set of categories that serves as data tags for the information you are collecting – Balance the number of fields you have compared to LOE – Don’t try to cram all of your tags into one field – Best to keep your tag options to <10 per field ▪ This is about standardizing the stuff you collect
- 21. Cataloging Examples- Reports, IOCs, CRM ▪ Triage threat reports into a system – Actor name(s), malware name(s), target(s), date of activity, date of report ▪ Parse IOCs into a spreadsheet or database – Sharing restrictions, STIX type, kill chain, date observed ▪ Contact List/ CRM solution – Relevant contact info, team, region, relationship – Relationship= supplier, partner, client
- 22. Cataloging Sample- Basic View FAKE CORRELATOIN & MARKINGS FOR DEMO PURPOSES ONLY
- 23. Cataloging Examples- Incident Response ▪ Cataloging data related to incident response include – Traditional SIEM functions (e.g. log consolidation and standardization) – # of systems impacted – Malware type & family name – IOCs identified (preferably cataloged by kill chain) – Level of Effort (LOE) for the analysts involved in the response
- 24. Cataloging Examples- Accounting for One-Offs ▪ Tracking one-offs and “time sucks” can be a pain ▪ Create a simple spreadsheet with – Date of activity – Analyst name – Category ▪ Meeting,Training, RFI,Admin, etc. ▪ Review your categories & data every few months to “tune” your system
- 25. Cataloging of the Future! (h/t Detect17) ▪ Do you track incident numbers tied to your IOC collections? ▪ Do you track the initial DETECTION point? ▪ Do you track which analysts identify the most incidents? ▪ Are you tracking costs over time?
- 26. Assess Phase ▪ If the first two steps are done correctly, this phase is very fast & fluid ▪ Assess Phase should lead to action, including – Deploying mitigations – Identifying additional data points to collect & catalog ▪ Analysis that leads to additional analysis is rarely useful – Fight the rabbit hole! – Fight the urge to analyze for the sake of analysis
- 27. Assess Phase- Basic Filters FAKE CORRELATOIN & MARKINGS FOR DEMO PURPOSES ONLY
- 28. Assessment Examples- Intel Methodologies ▪ Amazing analytical methodologies for intelligence analysis already exist! – “Assessment of Cause and Effect” – “RedTeam Analysis” (no, not that type of red team) – “Delphi Method” ▪ Recommend: Structured AnalyticalTechniques for Intelligence Analysts by Richard J. Heuer Jr. & Randolph H. Pherson
- 29. Assessment Examples- Technical Methodologies ▪ Examples of technical analysis in InfoSec – Malware analysis – Reverse engineering – Incident response ▪ Where are the methodologies for looking at technical analysis at an operational or strategic level?
- 30. Assess Phase for IR as Threat Analysts ▪ Technical/Tactical level assessments include – Type of attack (e.g. destructive, loss of IP, financial crimes) – Scope of impact to systems and recovery implications ▪ Operational level assessments include – Financial impact to the organization – Determining gaps in enterprise Defense in Depth ▪ Strategic level assessments that include – Actors intent, such as loss of IP for corporate espionage vs. political espionage
- 31. Assessment Example- Incident View FAKE CORRELATOIN & MARKINGS FOR DEMO PURPOSES ONLY
- 32. Assessment Example-Detection Point View FAKE CORRELATOIN & MARKINGS FOR DEMO PURPOSES ONLY
- 33. Act Phase ▪ Deploy mitigations – Signatures, rules, new tools, etc. ▪ Enforce policy change – Mandatory password reset – Require a new administrative policy and user training ▪ Publishing a final report – For your manager to take to a board of directors (*cough* MONEY) – As an information sharing report- maybe into a cool platform for sharing IOCs?
- 34. “Help me, help you!” CCAA as a Collaborative Model
- 35. Shared Models and Datasets ▪ CCAA as an information flow should be overlaid on top of your existing processes ▪ CCAA datasets should be accessible by multiple teams for greatest potential ▪ As the malware team is conducting technical analysis: – They identify the static observables (e.g. hashes, filename, file size, etc.) – They identify the malware family – They identify C2 observables AND push them to the CND team for signature creation and research against historical artifacts Collect Catalog Assess Act
- 36. Questions to Ask of Your Data ▪ A lot of malware calls out to “what is my IP” services – Has anyone asked “what valid applications need these services?” ▪ Is anyone looking at your passive DNS forThreat Intel? – Collect DNS logs,Catalog known sites, Assess unknowns,Act to block & further catalog
- 37. Are we there yet? Key takeaways & Resources
- 38. Did this guy just babble for an hour? ▪ Know your true environment: technical, operational, and strategic ▪ Understand how you support business operations and aim to ELi5 ▪ Spend more time cataloging & understanding your data to spend less time analyzing and chasing thoughts ▪ Stop sharing data! Start sharing information & exchanging knowledge
- 39. Final Thoughts ▪ We have to get better at creating repeatable & trainable processes – Medium to large companies- threat feeds can be leveraged for enrichment and pivoting – Small companies will have a greater challenge not drowning in data- get your processes firing tight & fast ▪ Analysis does not take dark magic nor special computers, but it does take experience and knowledge ▪ Only you can prevent computer fires
- 40. Hat Tips & Thanks ▪ The phia family for their support and shared experiences over the years ▪ Team Anomali and their dedication to connecting teams across the industry ▪ My current team and the stakeholders that I get to work with to share CTI
- 41. Resources From Along the Way ▪ “Former Federal CISCO on the Importance of FollowingThrough” – Gen. (Ret.) GregTouhill, NextGov, May 2017 ▪ “Passive DNS Monitoring –Why It’s Important for your IRTeam” – Phil Hagen, Red Canary, February 21, 2017 ▪ “Threat Intelligence at Microsoft: A Look Inside” – Sergio Caltagirone at SANS CyberThreat Intelligence Summit 2017 ▪ Little Bobby Comics & Robert M Lee’s efforts to mature CTI ▪ “APT Groups and Operations” – Google Drive…
Editor's Notes
- #4: Yes I have an agenda. Besides these fancy points, I’d also like to stop all the badness and may have a small Captain America complex. Of note, I consider it a Sergeant America complex because I worked for a living.
- #6: Sorry vendors, I kid, I kid. But seriously. We’ve all seen demos and bought tools and then failed in operations. ASK QUESTION Why do they fail?
- #7: The real reason they often “fail” is that we don’t deploy them fully/correctly for our environments because we don’t fully know our environments and no two networks are the same.
- #8: 7
- #10: Which one of these works better? Answer is “depends” on where you are sitting.
- #11: I was given an empty tent, a “Commanders Intent” (support medevac and VIP flights), and a few position titles. I spent the next few hours trying to explain to my boss how I would set up the environment to support the mission. Several hours because I kept getting it wrong. Through a few dozen attempts and a few hundred push-ups, I finally figured out what he was trying to teach me. We had a fixed position based on equipment requirements (radio) We had a basic understanding of our job titles and our mission We had me & a senior manager to work through the details
- #13: Discuss that all three are the same mapping designed for different conversations and different audiences.
- #14: 13
- #18: What are some of your regular tools for collecting data? Notepad Email And of course Office. I even picked up the nickname “Clippy” from a few coworkers. COLLECT ALL THE THINGS.
- #19: BAD EXAMPLES: Emailing reports and findings in large and frequent email blasts. I will autoroute your messages if you create noise in my life. Listen, you should review every email you send and ask yourself if it has a “so what”/action statement. If it doesn’t, it probably isn’t worth sending.
- #21: BAD EXAMPLES: Shared folders with dumps of reports in various formats. Stop trying to jam the square peg into the round hole
- #22: The importance of NULL vs. UNKNOWN and adding the statement “No further information available at the time of analysis”. Tell a 30 second vignette about malware URLs showing up clean one day and detected as malicious within a week’s end.
- #24: Yes, LOE is important for senior managers to understand. Collect & catalog that data for every incident and your annual budget is a lot easier to justify. You don’t want to send your manager or CISO into a budget meeting with just annual salary and licensing data. She’s gonna lose that argument for more resources!
- #26: DETECTION point Consider that we always focus on initial threat vector. What about tracking initial DETECTION point? This is how you could drive your resource discussions. Security costs money and the Board doesn’t like spending money without a direct return on their investment. We need to get better at having this conversation. One method I saw was a briefer calculated the average cost of all major breaches over the last few years and equated that an annual loss for that timeframe. If you are spending less than the average annual cost, then you are theoretically adding value. Again, there seems to be no agreed upon method between the financial folks and the security folks but that just means there is more work for us to do.
- #27: As we Assess, we may add and modify our existing collection mechanisms and update our set catalogs. What we must stop doing is recreating this cycle for every new event.
- #28: Use basic filters to narrow down the noise to what you are working on. Pro TIP for the TIP developers. ADD FILTERS TO ALL OF YOUR FIELDS THAT CAN TURN ON/OFF QUICKLY. Think Amazons UI.
- #29: There are a lot of great methodologies for the strategic level and some being built out for the operational level.
- #30: Explain the difference between technical/tactical, operational, and strategic.
- #31: We begin to look at events and threats in an attempt to assess impacts at the TECHNICAL/TACTICAL, operational, and strategic levels. Ops level is when we can apply that Continuity Planning math we’re all supposed memorize- you know “ALE” ARO” etc. But it also includes post-breach analysis. Understanding the impacts to business operations. Does your organization need to change its architecture or is just one tool/process that is missing? We need to get better and providing this QUALITATIVE ANALYSIS in the post-event discussion. IF we don’t do it as the analysts on the ground, then the bloggers of the world will surely fill that gap. Who’s message do we want here?
- #32: Filter to incident view to see a snapshot of what is known about that INC. Optionally- Filter by Detection Point = YES and find the type of IOC you are detecting the most.
- #33: Again, it may not be “sexy” for analysts, but it is crucial as a business case for new resources and “value added” to the organization. If you were in the metrics brief yesterday- HAT TIP John Holland and Paul Sheck from Team Anomali.
- #34: This section should be simple enough, but here we go These are the things that we do as analysts. We block/ mitigate/ clean up/ and-or write about our findings. Some systems are starting to automate these actions, as you have probably demo’d in the vendor area. If your TIP/SIEM/Cyber5000 appliance has “playbooks” to automate workflow- do you know what the system is doing? Do you know where those activities are being tracked in enterprise logs? Does the system leverage user or admin credentials and does your appliance use them securely? Going back to “system flow”- do you understand your systems? Can you ELI5?
- #35: Guatemala lag story with beer bottles as packets. “Literally my start to cyber training started in a hippie commune/ hostel place in Guat…” It is an example of collaboration and talking to your audience in terms they understand. This is far more powerful than using RFC and CVE numbers.
- #36: YARA RULE STORY: A truly intelligent threat analyst was writing a threat profile of an actor and pinged our malware team for the related IOCs. The malware team took the opportunity to explain how Yara works and how including that in report would be more beneficial than the ten pages of MD5s we were able to pull from our catalog system.
- #37: The malware team’s findings should be available and accessible DURING their assess phase. Remember that CCAA is cyclic and that CND is time sensitive.
- #42: General Touhill starts with a quote from Congressional Medal of Honor recipient Eddie Rickenbacker “There’s a six-word formula for success: Think things through, then follow through.” The General goes on to explain how many organizations buy too many tools they don’t need or properly use. There are a lot of great tips throughout this two part series. http://www.nextgov.com/technology-news/tech-insider/2017/05/former-federal-ciso-importance-following-through/137630/ This Red Canary blog is a simple method for turning DNS method into a threat intel source. https://www.redcanary.com/blog/passive-dns-monitoring-your-ir-team-needs-it/ Sergio points out that (loosely paraphrased): Most of your analytical power should be spent in collection and cataloguing your data. Analysis and action are executed faster and easier if your data is solid. https://youtu.be/1mxqwNgtNOE https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4Gx0YOIzlcBWMsdvePFX68EKU/edit#gid=1864660085