Exercise Your SOC: How to run an effective SOC response simulation (BSidesCharm 2018)
Download as PPTX, PDF0 likes181 views
The document outlines a comprehensive guide on conducting effective security operations center (SOC) response simulations, detailing the rationale, organization, and execution processes involved. It emphasizes the importance of validating tools and procedures through controlled attacks, establishing clear roles within the teams, and detailing the critical concepts and major steps necessary for effective exercises. Additionally, it highlights the need for post-exercise reviews to document findings and improve future security operations.
![Our Security Challenges
UNCLASSIFIED 6
Security: we are [usually] the last to know… and first to respond.](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-6-320.jpg)
![Why Do a SOC Exercise?
Validate your implemented TTPs for
SOC operate as expected
Test effectiveness of an attack
[in a controlled execution – on prod]
Finding your actual defense thresholds
UNCLASSIFIED 8](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-8-320.jpg)
![SOC Exercise Endgame
• Proactive training for attack patterns against your
organization’s TTPs [before advisory does]
• Learning together (red + blue) where your defenses &
processes are good, bad, and ugly and craft events together
• Determining where to focus [highly limited] resources post-
exercise to reduce risks of future compromise
UNCLASSIFIED 9
Avoid becoming a
[insert major news here]
article.](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-9-320.jpg)
![SOC Exercise: Critical Concepts
• Purpose is to be detected
–Run actions at low volume
below expected threshold
–Increase noise over time
until expected detection
–Plan for actions to turn the
dial up to 11 [w/o breaking all the things]
UNCLASSIFIED 12](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-12-320.jpg)
![SOC Exercise Selection: Examples
• Detect:
– When will we alert if [insert attack vector]
occurs to [insert system] at this time?
– Has [insert tool] been configured to
operational environment beyond defaults?
– What [insert tool(s)] will record an audit log
of this event’s observables?
UNCLASSIFED 18](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-18-320.jpg)
![SOC Exercise Selection: Examples
• Respond:
–Does [insert tool] respond as expected
to [insert attack vector]?
–Does SOC report on response capture
on [insert observables]?
–What [insert tool(s)] will audit log SOC’s
actions to response of the event?
UNCLASSIFED 19](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-19-320.jpg)
![SOC Exercise Selection: Examples
• Escalate:
–Did the exercise events escalate as
expected through [insert IR plan]?
–Are we meeting [insert IR plan]
response times?
–When [critical person in IR contact plan]
is out, what happens?
UNCLASSIFED 20](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-20-320.jpg)
![Determine Expected Outcomes: Measure
• Results of actions
– Should be able to map to STIX observable-
objects (previously CyBOX)
– Should [hopefully] create security events in
your security tooling for actions & reactions
– Be able to query in your SIEM [hopefully?]
UNCLASSIFIED 24
Be aware that SOC may be watching for your
recon as part of its monitoring ](https://image.slidesharecdn.com/exercise-your-socbsidescharm2018-190430125536/85/Exercise-Your-SOC-How-to-run-an-effective-SOC-response-simulation-BSidesCharm-2018-24-320.jpg)
Exercise Your SOC: How to run an effective SOC response simulation (BSidesCharm 2018)
- 1. Exercise Your SOC How to run an effective SOC response simulation 04/29/2018 Brian Andrzejewski, Senior Cyber Security Engineer Cyber Defense Branch, Information Security Division
- 2. Disclaimers • My personal views and opinions may not represent the position(s) of my employers (USCIS, DHS, or Federal Government) • Mention of any product names in this talk are not a government endorsement. • Questions? Raise hand to ask mid-stream! UNCLASSIFED 2
- 3. About Organization • USCIS: World’s largest immigration agency – Annually: • 8.5M applications received • 1M Permanent Residents • 730,000 new Citizens • 7,200 military personnel – Fee-funded • Over $4B in annual revenue • 17,000+ employees in over 200+ locations – We are a leading Federal agency in the cloud! UNCLASSIFIED 3
- 4. About Me • Cyber Defense - Lead Federal security engineer – Specialized in AppSec, DevSec, & CloudSec • Assess systems & designs for weaknesses & vulns • Build risk mitigations and “secure by default” profiles • Embrace “infrastructure-as-code” & rugged DevOps – IPT component rep for DHS Cloud Migration, DevSecOps, & Cyber Threat Intel – Advise Fed agencies on 2 yrs of InfoSec lessons learned in using Cloud, Agile, & DevOps methods • Prior life: – US Air Force / DoD Cyber Crime Center (DC3) • CNCI-5 / ESSA Program: Connect Fed Cyber Centers w/ STIX & TAXII • Workforce Dev: DC3 Digital Forensics Challenge / U.S. Cyber Challenge • Ops: DIBNET, Tool Assessment, Program Manager, Security Assessment – IT Dev & Ops: Commercial Healthcare, Consulting, & Academia UNCLASSIFIED 4
- 5. Personal Philosophies “Doveryai, no proveryai!” (trust, but verify) - Suzanne Massie “Attack is the secret of defense; defense is the planning of an attack.” – Sun Tzu Burn down your technical debt before it burns you. UNCLASSIFIED 5
- 6. Our Security Challenges UNCLASSIFIED 6 Security: we are [usually] the last to know… and first to respond.
- 7. Acronym Soup • APT: Advanced Persistent Threat • IR: Incident Response (typically SOC) • TTPs: Tools, Techniques, & Procedures • SOC: Security Ops Center Response team UNCLASSIFIED 7
- 8. Why Do a SOC Exercise? Validate your implemented TTPs for SOC operate as expected Test effectiveness of an attack [in a controlled execution – on prod] Finding your actual defense thresholds UNCLASSIFIED 8
- 9. SOC Exercise Endgame • Proactive training for attack patterns against your organization’s TTPs [before advisory does] • Learning together (red + blue) where your defenses & processes are good, bad, and ugly and craft events together • Determining where to focus [highly limited] resources post- exercise to reduce risks of future compromise UNCLASSIFIED 9 Avoid becoming a [insert major news here] article.
- 10. SOC Exercise: Team Roles • Red Leader – Has official comms to CISO on exercise execution – Leads team through exercise build, run, & closeout • Gold Leader – Monitors “chatter” for exercise IR detection response – Records outcomes to events executed • Execution Team – 1-2 Red Teamers specializing in the attack – 1-2 Blue Teamers specializing in defense config and its possible unexpected behaviors UNCLASSIFIED 10
- 11. SOC Exercise: Critical Concepts • Test hypotheses, not destroy targets – Objective testing of TTPs – Assume professionalism for targets involved – Use dummy data & modified, benign attacks – Let results show actual security posture UNCLASSIFIED 11 Question Recon Test Hypothesis Conclusion
- 12. SOC Exercise: Critical Concepts • Purpose is to be detected –Run actions at low volume below expected threshold –Increase noise over time until expected detection –Plan for actions to turn the dial up to 11 [w/o breaking all the things] UNCLASSIFIED 12
- 13. SOC Exercise: Critical Concepts • CISO owns risks to execute exercise – Picks estimated date time to execute – Give official green-yellow-red for exercise run – Owns the issues if things go sideways • Limit knowledge to exercise lifecycle – Keep “in the know” to least # of individuals – Space out recon from execution – like most APTs – Use trusted comms channels for exercise devOps (i.e. private chat “warroom” for recon, development, runtime, and reporting creation) UNCLASSIFIED 13
- 14. SOC Exercise Major Steps 1. Targeting Selection 2. Determine Expected Outcomes 3. Brief CISO to Signoff 4. Exercise Day 5. Lessons Learned Debrief UNCLASSIFIED 14
- 15. Targeting Selection • State a hypothesis to form a conclusion “When I run attack A against asset Y, then: – Defense D should trigger – Process P should happen – Response R should occur • Recon your target’s observables for ops and risks – Leverage historical logs for events to timeline analysis – Determine target’s IT & business ops lows and highs – Research target’s dependencies and if outside TTPs are dependent on it UNCLASSIFIED 15
- 16. Targeting Selection: Crawl-Walk-Run • Use Crawl-Walk-Run Approach – Use laser focus by leveraging existing data – Target improving missional critical processes – Honor the process to build the trust • Establishes a track record of success • Use to obtain immediate results – and use again to verify lessons learned were implemented UNCLASSIFIED 16
- 17. Targeting Selection: Crawl-Walk-Run (cont.) • Crawl – Focus on a simple security rule or past IR – Example: “If I run an exercise actions below, at, and above a known alerting behavior, what should happen?” • Walk – Coordination w/ others outside of immediate security team – Example: Explicit permission to emulate a particular user for exercise execution in a mission critical process. • Run – Complex exercises once trust is fully established – Example: Multi week exercise to establish footholds and pivot to critical datasets UNCLASSIFIED 17
- 18. SOC Exercise Selection: Examples • Detect: – When will we alert if [insert attack vector] occurs to [insert system] at this time? – Has [insert tool] been configured to operational environment beyond defaults? – What [insert tool(s)] will record an audit log of this event’s observables? UNCLASSIFED 18
- 19. SOC Exercise Selection: Examples • Respond: –Does [insert tool] respond as expected to [insert attack vector]? –Does SOC report on response capture on [insert observables]? –What [insert tool(s)] will audit log SOC’s actions to response of the event? UNCLASSIFED 19
- 20. SOC Exercise Selection: Examples • Escalate: –Did the exercise events escalate as expected through [insert IR plan]? –Are we meeting [insert IR plan] response times? –When [critical person in IR contact plan] is out, what happens? UNCLASSIFED 20
- 21. Determine Expected Outcomes 1. Develop timeline of major actions 2. Quantify possible risks to execution 3. Establish measurements to actions UNCLASSIFIED 21
- 22. Determine Expected Outcomes: Actions • DO – Execute attack below know defense thresholds – Build actions to increase over time to & exceed known thresholds – Plan for execution failure failbacks • DON’T – Execute an attack you have not tried in a offline test environment version first – Always assume an attack will be successful UNCLASSIFIED 22
- 23. Determine Expected Outcomes: Risks • Identify Overall Risks to Ops – Critical operations at peak use – Execution during other incidents – Breaking an unrecoverable system – Unknown interdependencies • Quantify Probability & Impacts for Events UNCLASSIFIED 23 Action Probably Impact Risks Threshold T triggers for Observable O1 in SOC tool S1 High Low Recon identified manufacturer defaults by Method M1
- 24. Determine Expected Outcomes: Measure • Results of actions – Should be able to map to STIX observable- objects (previously CyBOX) – Should [hopefully] create security events in your security tooling for actions & reactions – Be able to query in your SIEM [hopefully?] UNCLASSIFIED 24 Be aware that SOC may be watching for your recon as part of its monitoring
- 25. Determine Expected Outcomes: Measure • Use objective measurements – i.e. – Time between planned actions vs. actual – Time taken to execute action (start to finish) – Frequency of action – Successful/failure of action • Prepare for *other* findings – Unusual output to action that was not expected – Discovering other possible defense weaknesses – Ensuring documented disclosure post- exercise UNCLASSIFIED 25
- 26. Briefing CISO to Signoff • Be clear on purpose of the exercise “Evaluating our DLP sensors at X location on our network will detect PII exfiltration to policy Y” • Explain why this is important – i.e. – Validating that Tool X is configured to policy Y – Testing Response X performs A, B, & C actions – Verifying Threshold H will trigger under these conditions. UNCLASSIFIED 26
- 27. Briefing CISO to Signoff (cont.) • Discuss action plan and its risks – CISO must be able to communicate to their bosses if something goes wrong – Confirm accuracy to business & IT ops expectations from recon observables – Adjust to CISO’s expectations • Establish parameters for when to start/stop – Business impacts – IT Ops impacts – Actions exceed expected thresholds UNCLASSIFIED 27
- 28. CISO Signoff (cont.) • Establish exercise agreement in writing – This is your “get out of jail free card” – Require CISO to sign physically or digitally – If physically testing, carry on-person • Explicitly obtain exercise day(s) w/ CISO – Expect date to change based on ops rhythms – Establish method of comms prior to “go-time” UNCLASSIFIED 28
- 29. Exercise Day • Preparation – Comms channels for WAR online war room – Team in channel and prepared for run – CISO gives Red Leader green light in writing • Runtime – CISO in channel to watch progress of activity – Report event action progress in channel – Immediately report any seen unusual observables – Explicitly report when TTP thresholds exceeded before pumping the volume up to 11 UNCLASSIFIED 29
- 30. Exercise Day: Measure Actions • Record objective findings for events – Event outputs – Changes in timetable for event execution – Individuals responding to event execution • Document unexpected outcomes – i.e. – Attack did not execute successfully – Attack exposed an additional, out-of-scope vulnerability UNCLASSIFIED 30
- 31. Exercise Day: End of Exercise • Confirm with CISO on exercise closure CISO may want opportunity to test an additional action in scope if undetected • Hold post-op call with team + CISO – Collect thoughts on runtime results – Document any lessons learned during runtime – Discuss anything that was unexpected UNCLASSIFIED 31
- 32. Lessons Learned: Initial Report • Create briefing on exercise – Audience – minimum: • Exercise team + CISO • Person(s) impacted by exercise testing • Assume exec level – Provide purpose of exercise – Show execution timeline for expected to actual results – Keep details on tap for live briefing for Q&A UNCLASSIFIED 32
- 33. Lessons Learned: Initial Report • Briefing – be interactive! – Maintain blameless environment (just the facts!) – Solicit feedback from those targeted in exercise – Propose possible solutions to mitigate or correct for discussion points UNCLASSIFIED 33
- 34. Lessons Learned: Post Report • Provide actionable goals and timelines to resolve findings – Leverage input from targets of exercise – Quantify with results of exercise • Build goals into performance and risk management plans – Project development or maintenance tasks – Adjusting existing monitoring controls – Documenting into security plans for auditors UNCLASSIFIED 34
Editor's Notes
- #2: For FY17 version of slides, add new 1st Slide on Past Year results