Secure360 on Risk
2 likes832 views
The document critiques the current state of risk management in information security, emphasizing the reliance on poor quality data and inadequate models that hinder effective decision-making. It advocates for an evidence-based approach to improve data quality and execution in risk management, while suggesting that security should be viewed as an emergent property of complex systems rather than isolated components. The authors propose using structured methodologies like the GQM approach to better define goals and metrics, ultimately leading to more effective security outcomes.
Secure360 on Risk
- 1. Challenging Conventional Wisdom: A New Approach to Risk Management Alex Hutton Jay Jacobs
- 2. What’s this We think you’re getting bad information! about? We think our industry can do better! We think this will make us “more secure!”
- 3. Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer
- 4. How are you making decisions now?
- 5. What’s the quality of those decisions?
- 6. Effective Decisions need quality data, models, execution
- 7. Our vendors and standards aren’t helping us (-:
- 8. hey, why are you getting lousy information from standards and vendors?
- 9. The science hey, why are of information you getting security & risk management lousy is hard information 1. Pseudo Science & Proto Science from 2. Models & Data standards 3. Complexity and vendors?
- 10. The science hey, why are of information you getting security & risk management lousy is hard information 1. Pseudo Science & Proto Science from 2. Models & Data standards 3. Complexity and vendors?
- 11. State of the Industry (a) (Thomas Kuhn is way smarter than we are) proto-science somewhat random fact gathering (mainly of readily accessible data) a“morass”of interesting, trivial, irrelevant observations a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gathering
- 12. State of the Industry (b) At our present skill in measurement of security, we generally have an ordinal scale at best, not an interval scale and certainly not a ratio scale. In plain terms, this means we can say whether X is better than Y but how much better and compared to what is not so easy. – More from Dan Geer
- 13. If Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, how about InfoSec? Where do we sit in the family of sciences?
- 14. We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.
- 15. Take, for example, CVSS
- 16. “the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”
- 17. Jet Engine X Peanut Butter = Shiny
- 18. decimals aren’t magic. adding one willy-nilly doesn’t suddenly transform ordinal rankings into ratio values.
- 19. The science hey, why are of information you getting security & risk management lousy is hard information 1. Pseudo Science & Proto Science from 2. Models & Data standards 3. Complexity and vendors?
- 20. Data must exist in order to feed our models... ... but creating the right models are dependent on understanding what data is useful! 20
- 21. Data, Models, Execution: Garbage in-Garbage Out
- 22. Data, Models, Execution: Treat Data Poorly
- 23. Data, Models, Execution: Adapting to Situations
- 24. The science hey, why are of information you getting security & risk management lousy is hard information 1. Pseudo Science & Proto Science from 2. Models & Data standards 3. Complexity and vendors?
- 25. These “risk” statements you’re making... I don’t think you’re doing it right. - (Chillin’ Friederich Hayek)
- 27. A Comforting Thought... “Given Newton's laws and the current position and velocity of every particle in the universe, it was possible, in principle, to predict everything for all time.” -- Simon-Pierre LaPlace, 1814
- 28. 8 4 4 2 2 2 2 Reductionism
- 29. 8 ? 4 4 ? 2 2 2 2 Functionalism
- 30. Asset Reductionism Functionalism Comp. Comp. Sub. Sub. Attribute Attribute Attribute Attribute
- 31. Awww man... ...even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately. ... small differences in the initial conditions produce very great ones in the final phenomenon. A small error in the former will produce an enormous error in the latter. Prediction becomes impossible... -- Henri Poincare, 1887
- 32. ty non lexi -l i p nea C om r 13 5 6 2 2 2 2 Systems Approach Holism
- 33. Complex systems contain changing mixtures of failures latent within them. The complexity of these systems makes it impossible for them to run without multiple flaws being present. ... individually insufficient to cause failure ...failures change constantly because of changing technology, work organization, and efforts to eradicate failures. Complex systems run in degraded mode. “How Complex Systems Fail” - Richard Cook
- 34. Security is a characteristic of systems and not of their components Security is an emergent property of systems; it does not reside in a person, device or department of an organization or system. ... it is not a feature that is separate from the other components of the system. ...the state of Security in any system is always dynamic “How Complex Systems Fail” - Richard Cook
- 35. We may want to rethink our approach.
- 36. Overcoming the problem • Medicine uses an “Evidence- Based” approach to solving problems in the complex system that is the body. • Dr. Peter Tippett (MD, PhD) applies Evidence-Based principles to Information Security. 36
- 37. What to study: Sources of Knowledge Suggested context: Capability to manage (skills, resources, asset decision quality…) landscape impact landscape risk threat landscape controls landscape
- 38. How: Data Quality in Evidence-Based Practice Evidence level D Evidence level C Evidence level B Evidence level A Evidence level A Case-‐series Consistent Consistent “Expert opinion study or Retrospec8ve Randomized without explicit extrapola8ons Cohort, Exploratory Controlled Clinical cri8cal appraisal, from level B Cohort, Ecological Trial, cohort study, or based on studies. Study, Outcomes all or none, clinical physiology, bench Research, case-‐ decision rule research or first control study; or validated in principles.” extrapola8ons from different level A studies. popula8ons. beNer
- 39. Evidence-Based Risk Management State of Nature State of Knowledge State of Wisdom Evidence level D Lists Feeling like we’ve done something Evidence level C Simple derived values Outcomes with ad-hoc with ad-hoc modeling deductive selections Evidence level B Formal Modeling Decision making constructs Evidence level A
- 40. Evidence-Based Risk Management State of Nature State of Knowledge State of Wisdom Evidence level D Lists Feeling like we’ve done something Evidence level C Simple derived values Outcomes with ad-hoc with ad-hoc modeling deductive selections Evidence level B Formal Modeling Decision making constructs Evidence level A
- 41. Evidence-Based Risk Management State of Nature State of Knowledge State of Wisdom Evidence level D Lists Feeling like we’ve done something Evidence level C Simple derived values Outcomes with ad-hoc with ad-hoc modeling deductive selections You are here Evidence level B Formal Modeling Decision making constructs Evidence level A
- 42. So How Do We Change? Data Models… Standards START WITH THE OUTCOMES!
- 43. Two True Security Outcomes: Success and Failure
- 44. Knowing Success in InfoSec is hard - Known Success (anti-Threat ops) - Unknown success (controls work without us knowing) - Dumb luck (We’re not targeted, but our neighbor is)
- 46. Getting the outcomes: Success stronger processes result in fewer availability incidents
- 47. Getting the outcomes - Successes: - Existences of processes - Operational (performance) metrics - Maturity ratings WHAT WE WANT ARE PATTERNS!
- 48. Knowing Failure is (somewhat) easier
- 49. Getting The Outcomes: Failures VERIS | Verizon Enterprise Risk and Information Sharing VERIS takes the incident narrative and creates metrics (risk determinants)
- 50. VERIS | Verizon Enterprise Risk and Information Sharing A free (as in beer*) framework created for metrics, modeling, and compara8ve analy8cs. A security incident (or threat scenario) is modeled as a series of events. Every event is comprised of the following 4 A’s: Agent: Whose acLons affected the asset AcLon: What acLons affected the asset Asset: Which assets were affected AOribute: How the asset was affected
- 51. VERIS takes this : INCIDENT REPORT “An attacker from a Russian IP address initiated multiple SQL injection attacks against a public-facing web application. They were able to introduce keyloggers and network sniffers onto internal systems. The keyloggers captured several domain credentials which the attackers used to further infiltrate the corporate network. The packet sniffers captured data for several months which the attacker periodically returned to collect…” and…
- 52. …and translates it to this… Event 1 Agent: External (Org crime) Action: Hacking (SQLi) Asset: Server (Web server, Database) Attribute: Integrity Event 2 Agent: External (Org crime) Action: Malware (Keylogger) 1 > 2 > 3 > 4 > Asset: Server (Web server) Attribute: Confidentiality Event 3 Agent: External (Org crime) Action: Hacking (Use of stolen creds) Asset: Server, Network (multiple) Attribute: Confidentiality, Integrity Event 4…
- 54. patterns!
- 55. Framework = ∑ ∩ ∫√ Models Data
- 56. Framework Framework Data Process = Process ∑ ∩ ∫√ Models = ∑ ∩ ∫√ Data Models Process Process Data
- 57. Using your metrics program - Identify & Measure your processes - Identify & Measure your failures - Get into loss factors (ABC) - Share data - Support data sharing efforts
- 58. Bring it Home: your metrics program
- 59. Bring it Home: your metrics program or
- 60. Bring it Home: your metrics program or The Amazing Technicolor Scorecard
- 61. Priority #1: no more surrogate data
- 62. Priority #1: (meaning) no more risk analysts*
- 63. Priority #1: (really) create data analysts
- 64. Data analysts need to focus on quality data, models, execution
- 65. Evidence-Based Risk Management State of Nature State of Knowledge State of Wisdom Evidence level D Lists Feeling like we’ve done something Evidence level C Simple derived values Outcomes with ad-hoc with ad-hoc modeling deductive selections Evidence level B Formal Modeling Decision making constructs Evidence level A
- 66. asset landscape A balanced scorecard of sorts threat impact landscape landscape risk controls landscape
- 67. Where to look? The Two True Security Outcomes: Success and Failure
- 68. Failures: threat landscape incidents, red/blue team asset vulnerabilities, misconfigurations, landscape unknowns... gaps in coverage, known lack of controls landscape effectiveness, known underskilled/ utilized... impact Cost-Based Accounting around landscape incidents, cost of operations, etc...
- 69. Successes: threat landscape intel, red/blue teams, SIEM asset vulnerabilities, misconfigurations, landscape unknowns, skills, training controls positive threat outcomes (tOps), skills, landscape training impact landscape ROI? ROSI? (ducks to avoid tomatoes)
- 70. What to look? Two types of data to find: Focus initially on Visibility, then look to find Variability.
- 71. How to look? The GQM Approach: For each “where” for each “what” use the following “how”
- 72. How to look? The GQM Approach: For each “where” for each “what”, start by using GQM as “how.”
- 73. Goal, Question, Metric Conceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view. Operational level (question) questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal. Quantitative level (metric) Victor Basili metrics, based on the models, is associated with every question in order to answer it in a measurable way.
- 74. The Book You Should Buy (Jay & Alex aren’t getting a kickback, in case you’re wondering)
- 75. GQM for Fun & Profit Goals establish what we want to Goal 1 Goal 2 accomplish. Questions help us understand how to meet the goal. They Q1 Q2 Q3 Q4 Q5 address context. Metrics identify the measurements that are needed to answer M1 M2 M3 M4 M5 M6 M7 the questions.
- 76. GQM for Fun & Profit Execution Goal 1 Goal 2 Models Q1 Q2 Q3 Q4 Q5 Data M1 M2 M3 M4 M5 M6 M7
- 77. data about defined success and failures models of assets, controls, threats contributing to impact execution by data analysts ...Feeding standards, audits and governance
- 78. Using your metrics program - Identify & Measure your processes - Identify & Measure your failures - Get into loss factors (ABC) - Share data - Support data sharing efforts
- 79. Using your metrics program - Identify & Measure your processes - Identify & Measure your failures - Get into loss factors (ABC) - Share data - Support data sharing efforts
- 80. Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer
- 81. Questions? Jay Jacobs Alex Hutton @jayjacobs @alexhutton jay@beechplane.com alex@alexhutton.com
- 82. Approaching the system as a system asset landscape impact Prioritize landscape risk threat landscape controls landscape De-prioritize
- 83. Suggested context: Capability to manage (skills, resources, decision quality…) asset landscape impact landscape risk threat landscape controls landscape
- 84. Data Sharing: - Sources: - Qualify this Intel according to framework - Treat with appropriate data quality listings (let models shape the certainty)
- 85. Get Into Accounting - Use existing models that take advantage of accounting concepts (ABC) to Talk to the LOBs
- 86. Using your metrics program - Identify & Measure your processes - Identify & Measure your failures - Share data - Support data sharing efforts - Get into loss factors (ABC)
- 87. Challenging Conventional Wisdom Conventional Wisdom may not be wrong - Question current practices - Seek Evidence and Feedback