Confirmation Bias - How To Stop Doing The Things In IT Security That Don't Work
1 like822 views
This document summarizes a presentation about overcoming confirmation bias in security. The presentation discusses how security professionals often rely on metrics that don't provide useful information for managing risk. It advocates developing quantitative risk scores linked to business goals to make better risk management decisions. The presentation also warns against signs of confirmation bias, like only looking at past security events rather than probability of vulnerabilities, and provides tips for creating effective security metrics.
Confirmation Bias - How To Stop Doing The Things In IT Security That Don't Work
- 1. Copyright © 2010-2011 IANS. The contents of this presentation are confidential . All rights reserved. Confirmation Bias How to Stop Doing the Things in Security That Don't Work November 2011
- 2. 2Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Who am I? » Michael A. Davis – CEO of Savid Technologies • IT Security, Risk Assessment, Penetration Testing – Speaker • Blackhat, Defcon, CanSecWest, Toorcon, Hack In The Box – Open Source Software Developer • Snort • Nmap • Dsniff » Savid Technologies – Risk Assessments, IT Security Consulting, Audit and Compliance
- 3. 3Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Author
- 4. 4Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. The Issue “Single biggest security related problem is a lack of Senior Level commitment to enterprise wide security policies.“ Source: 2011 InformationWeek Strategic Security Survey, June 2011
- 5. 5Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Execs Are Paying Attention 0% 5% 10% 15% 20% 25% 30% 35% 40% Exec Involvement Budget Constraints 2010 2011 Source: Information Week Data Survey, 2011
- 6. 6Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. We Protect, They Are Criticized According to Bloomberg News, Sony has been subpoenaed by New York attorney general Eric Schneiderman, who is "seeking information on what Sony told customers about the security of their networks, as part of a consumer protection inquiry." (Source: informationweek.com) Rep. Mary Bono Mack (R-Calif.), the subcommittee chair, said that Sony should have informed its consumers of the breach earlier and said its efforts were “half-hearted, half-baked.” She was particularly critical of Sony’s decision to first notify customers of the attack via its company blog, leaving it up to customers to search for information on the breach. (Source: washingtonpost.com)
- 7. 7Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. We All Do Them Source: 2011 InformationWeek Analytics Strategic Security Survey 0% 10% 20% 30% 40% 50% 60% 70% 80% Yes No Don't Know % that perform Risk Assessments 2011 2010
- 8. 8Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. The Reality Source: 2011 InformationWeek Analytics Strategic Security Survey Very 30% Somewhat 67% Not At All 3% Risk Assessment Effectiveness
- 9. 9Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Complex IT Projects Fail - A lot Out Of 200 Multi-nationals: 67% Failed To Terminate Unsuccessful Projects 61% Reported Major Conflicts 34% Of Projects Were Not Aligned With Strategy 32% Performed Redundant Work 1 In 6 Projects Had A Cost Overrun Of 200%! Source: 2011 Harvard Business Review – Berlin Univ Technical survey
- 10. 10Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. T-Mobile CISO On Metrics “Security experts can't measure their success without security metrics, and what can't be measured can't be effectively managed.” ~ Bill Boni, VP of IS, T-Mobile USA
- 11. 11Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Why Do We Care? Management Asks: –“Are We Secure?” Without Metrics: –“Depends How You Look At It” With Metrics: –“Look At Our Risk Score Before This Project, It Dropped 15%. We Are More Secure Today Than Yesterday”
- 12. 12Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Metrics, We need metrics!
- 13. 13Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Where/What to measure Strategy/Governance Code Reviews, Project Risk Assessments, Exceptions/Waivers Tactical/Sec Ops Vuln Management, Patch Management, Incidents, etc. IS Budget Spending/employee Policy gaps in existence Industry Standards Adopted Awareness Plan % projects going through assessment process # of policy exceptions # of risk acceptances % project doing code reviews Error rates Freq of vuln assessment # outstanding vulns Rate of fixing Trend of incident response losses
- 14. 14Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Who are you? TCO Patch Latency SPAM/AV Stats
- 15. 15Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Examples of metrics Baseline Defenses Coverage (AV, FW, etc) – Measurement of how well you are protecting your enterprise against the most basic information security threats. – 94% to 98%; less than 90% cause for concern Patch Latency – Time between a patch’s release and your successful deployment of that patch. – Express as averages and criticality Platform Security Scores – Measures your hardening guidelines Compliance – Measure departments against security standards – Number of Linux servers at least 90% compliant with the Linux platform security standard
- 16. 16Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Phishing Still Works
- 17. 17Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Stop With The Confirmation Bias Risk Perception Is Bad –Tornado V. Kitchen Fire –Less Familiar Are Perceived As Greater Risk Favor Info That Match Preconceptions Cause And Effect Processing Correlation Does Not Equal Causation We Manage Risk Using Metrics That Don’t Matter
- 18. 18Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved.
- 19. 19Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. The Formula Of Successful Risk Management PBL = λ1 x p1 + λ2 x p2 + λ3 x p3
- 20. 20Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Hazard vs. Speculative Risk
- 21. 21Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Linking to Business Goals Copyright Carnegie Mellon SETI MOSAIC Whitepaper
- 22. 22Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Outcome Management Copyright Carnegie Mellon SETI MOSAIC Whitepaper
- 23. 23Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. It Is About Risk MANAGEMENT Effective Metrics Catalog Define: Category Metric How To Measure Purpose Of This Metric Target Audience Reporting Frequency/Period
- 24. 24Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. 5 Signs You Have a Confirmation Bias Using Quantitative Risk Scores To Make Decisions Look At Security Events Instead Of Probability Of Vulnerabilities Talk About Risk In Terms Of “Industry Data” Lack Of Risk Management Inability To Communicate Risk
- 25. 25Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Security Metric Gotchas Not Tracking Visibility –What % is the metric representing? –Develop baseline for acceptance Not Trending –Provide at least 4 previous periods and trend line Not Providing Forward Guidance –Red, Green, Yellow (Worse, Better, Same) Not Mapping To A Business goal Focusing on Hazard Risk Not Using Qualitative Metrics
- 26. 26Copyright © 2010-2011 IANS. The contents of this presentation are confidential and may not be distributed without IANS’ permission. All rights reserved. Contact Information Michael A. Davis mdavis@savidtech.com 708-532-2843 Twitter: @mdavisceo