Presenting Metrics to the Executive Team

Session ID:
Session Classification:
John D. Johnson
Security Strategist
Presenting Metrics
to the Executive Team
SEM-003
Intermediate

Questions:
 How do we define security metrics?
 How are security metrics useful?
 Where do get the information, and how do we
turn it into something meaningful?
 How do we present security metrics to our
management?
 Building a security metrics program
 Group Discussion: What works for you?
2

Measurements & Metrics
 Performance metrics measure how well an
organization performs
 Drives process improvements and demonstrates
value-add
 Metrics can show how we compare to our peers
 Metrics can help us break out of the cycle that
comes from relying on products from vendors to
rescue us from new threats:
Detect  Report  Prioritize  Remediate
4

Security Metrics
 Make security metrics more meaningful to
stakeholders
 We need to learn to ask the right questions, if our
results are going to be meaningful
 The best metrics are SMART: Specific, Measurable,
Attainable, Repeatable & Time-Dependent
 This is an inherently difficult problem
 What is meaningful to stakeholders?
 Can we make metrics more quantitative?
 What can we measure?
 What are our peers doing?
5

Motivations
 Various Motivations for Developing Metrics
 Regulations - Compliance
 Audits (both internal and external)
 Money (security is rarely a profit center)
 Responding to new threats
 Enabling new technology and business processes
 Awareness: Making executives aware of trends
 Example Compliance Metrics:
 Manager sign-off on access controls
 A&A control artifacts
 Audit reports/findings (number, severity, BU)
 Exception reporting/tracking
 PCI Compliance status, dates
6

Example Security Metrics
 Application Security
 # Applications, % Critical Applications, Risk Assessment Coverage, Security Testing
Coverage
 Configuration Change Management
 Mean-Time to Complete Changes, % Changes w/Security Review, % Changes w/Security
Exceptions
 Financial
 Infosec Budget as % of IT Budget, Infosec Budget Allocation
 Incident Management
 Mean-Time to Incident Discovery, Incident Rate, % Incidents Detected by Controls, Mean-
Time Between Security Incidents, Mean-Time to Recovery
 Patch Management
 Patch Policy Compliance, Patch Management Coverage, Mean-Time to Patch
 Vulnerability Management
 Vulnerability Scan Coverage, % Systems w/o Known Severe Vulnerabilities, Mean-Time to
Mitigate Vulnerabilities, # Known Vulnerability Instances
7
* Source: Center for Internet Security

Gathering Data
 Data can be qualitative or quantitative
 Data can be coarse-grained or fine-grained
 Data can involve ordinal or cardinal numbers
 Less mature programs often have historical data to use
 Coarse-grained, qualitative, requires interpretation
 Examples: Audit findings, incident reports, viruses…
 More mature programs use multiple data sources
 Data from different sources can provide context, it is
important to consider the type of meta data that can
be gathered to add value later on
8

Modeling Data
 Some good standard assessment frameworks
can be used to provide a standard taxonomy for
describing risk
 Common frameworks allow data to be shared
and compared between companies
 Good models allow better analysis of complex
risk scenarios
 Examples: CAPEC, FAIR and VERIS
 Example of Industry Data: Verizon DBIR
9

Operational, Tactical & Strategic Metrics
 Operational plans lead to accomplishing tactical
plans, which in turn lead to accomplishing
strategic plans (which in turn are aligned with
business objectives).
 Tactical & Operational: IDS, Forensics, Help
Desk Tickets, Time to Patch, Viruses Blocked,
Support, Change Management…
 Strategic Metrics: Overall Compliance,
Compared to Baseline, Identifies Gaps in
Program, Shows Business Alignment & Value
10

Learn Where Others Succeed & Fail
11
 Successful security leaders overcome
confirmation bias and compare notes more often
with peers
 Standards and frameworks help a company
establish a baseline
 Results need to be translated into a context that
is relevant for your business
 Be aware that executives may downplay the
significance of industry data and feel their
company is the exception to the rule

Good or Bad?
13
© Pedro Monteiro of the What Type blog

Good or Bad?
18
Applied Security Visualization, Raffael Marty

Good or Bad?
19
Applied Security Visualization, Raffael Marty

Good or Bad?
20
http://www.pentest-standard.org

Security Metrics for Management
 Find a way to add business value
 Meeting regulatory requirements
 Consolidation of tools, reduction of resources
 Demonstrate reduced costs by reduction in help desk cases
 Business leaders take the loss of IP seriously
 Have security seen as a business enabler. New technologies
come with risks, but they may also lead to new innovations and
competitive advantage.
 Explain it in language business leaders understand
 Make presentations clear & concise
 Avoid IT jargon
 Provide the information executives need to make informed
decisions
23

Building a Security Metrics Program
 Decide on your goals and objectives at the onset
 Long-term and short-term goals
 Identify key metrics (SMART) to generate
 Will these be qualitative or quantitative?
 Will these be manual or automated?
 Will these be based on a standard framework, or vetted against peers, or use
some other model?
 Will these be tactical, operational, strategic or business metrics?
 Establish a baseline and targets
 Determine how best to present metrics in a consistent way, for
audience and frequency
 Get stakeholder buy-in and feedback; deliver balanced scorecard
 Develop a process for continuous improvement
24

References
 CAPEC, http://capec.mitre.org
 Verizon DBIR, http://www.verizonbusiness.com/go/2011dbir
 Verizon VERIS Framework, https://www2.icsalabs.com/veris/
 FAIR Framework, http://fairwiki.riskmanagementinsight.com/
 Center for Internet Security, Security Metrics, http://benchmarks.cisecurity.org/en-
us/?route=downloads.metrics
 Trustwave SpiderLabs Global Security Report, https://www.trustwave.com/GSR
 Ponemon Institute, http://www.ponemon.org
 Security Metrics: Replacing Fear, Uncertainty, and Doubt, Andrew Jaquith (2007)
 Metrics and Methods for Security Risk Management, Carl Young (2010)
 Security Metrics, A Beginner’s Guide, Caroline Wong (2011)
 Applied Security Visualization, Raffael Marty (2008)
 The Visual Display of Quantitative Information, Edward Tufte (2001)
25

References
 New School Security Blog, http://newschoolsecurity.com/
 SecurityMetrics.org, http://securitymetrics.org/
 A Few Good Metrics, http://www.csoonline.com/read/070105/metrics.html
 Measuring Security, Dan Geer, http://geer.tinho.net/measuringsecurity.tutorial.pdf
 CIS Consensus Security Metrics v1.0.0,
https://community.cisecurity.org/download/?redir=/metrics/CIS_Security_Metrics_v1.0
.0.pdf
 Performance Measurement Guide for Information Security,
http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf
 Directions in Security Metrics Research, http://csrc.nist.gov/publications/drafts/nistir-
7564/Draft-NISTIR-7564.pdf
 A Guide to Security Metrics,
http://www.sans.org/reading_room/whitepapers/auditing/a_guide_to_security_metrics
_55
 Patch Management and the Need for Metrics,
http://www.sans.org/reading_room/whitepapers/bestprac/1461.php
26

References
 The Security Metrics Collection,
http://www.csoonline.com/article/455463/The_Security_Metrics_Collection
 Implementing a Network Security Metrics Program,
http://www.giac.org/certified_professionals/practicals/gsec/1641.php
 Choosing the Right Metric, http://www.juiceanalytics.com/writing/choosing-rightmetric/
 Web Metrics Demystified, http://www.kaushik.net/avinash/2007/12/webmetrics-
demystified.html
 Blogs about: Security Metrics, http://en.wordpress.com/tag/security-metrics/
 Standardizing metrics and their presentation,
http://www.unifiedcompliance.com/it_compliance/metrics/reporting_standards/standar
dizing_metrics_and_thei.html
 Getting to a Useful Set of Security Metrics,
http://www.cert.org/podcast/show/20080902kreitner.html
 Dashboards by Example, http://www.enterprise-dashboard.com/
 Excel Charting Tips, http://peltiertech.com/Excel/Charts/index.html
27

Presenting Metrics to the Executive Team

More Related Content

What's hot (20)

Similar to Presenting Metrics to the Executive Team (20)

More from John D. Johnson (13)

Recently uploaded (20)

Presenting Metrics to the Executive Team

Editor's Notes