Metrics & Reporting - A Failure in Communication

Metrics & Reporting 2
CONTENTS
Metrics and Reporting ............................................................................................................. 3
The Problem Measured........................................................................................................... 4
Is This Important? ................................................................................................................... 5
‘Communication is What the Receiver Does’ .......................................................................... 6
What IT/Security is Doing........................................................................................................ 7
The Danger in Poor Communication..................................................................................... 10
What is IT/Security Doing About this Lack of Communication? ............................................ 10
What Should IT/Security Be Doing?...................................................................................... 11
© 2014 Wisegate. All Rights Reserved. All information in this document is the property of Wisegate.
This publication may not be reproduced or distributed in any form without Wisegate's prior written
permission. There’s a good chance we’ll let you use it, but still: it’s nice to ask first.

a Failure in Communication 3
In June of 2014, Wisegate conducted a member-driven research initiative designed to
assess the current state of security risks and controls in business today. Assessing IT
Security Risks addresses many of the top takeaways from this survey. This document is the
first in a series of reports designed to look more closely at four specific issues highlighted
by that survey.
» Metrics and reporting
» Malware and data breaches
» Data-centric security
» Automation and orchestration
Metrics and Reporting
This document might have just as easily been titled, ‘The Lack of Metrics’. It is highlighted
in a simple conclusion reached in Assessing IT Security Risks:
“Overall, [security] teams were optimistic but not overwhelmingly confident.”

On the surface, this statement appears to hide a contradiction: how can someone be
genuinely optimistic without being simultaneously confident? That apparent contradiction
hides a potentially widespread problem in information security: CISOs are always improving
their company security; there is little ability, however, to measure that success (or indeed,
lack of it).
Without having the metrics of success or failure, security teams can be optimistic in what
they are doing—but cannot ultimately be confident in its effect.
This problem is then compounded. Metrics form the basis of business-level reporting, and
without those metrics IT struggles to effectively communicate security issues to Business.
The Problem Measured
Participants in this survey were asked, ‘do you have metrics in place to track your top three
risks?’ (see Figure 1). Overall, 50% do not have metrics.
…the real problem with security risk management in the enterprise isn’t of
confidence—it’s of measurement; survey respondents don’t really have a good way
of indicating the effectiveness (or lack thereof) of existing programs.
—Assessing and Managing IT Security Risks
Figure 1: Survey Question: Do you have a metric to measure the risk in your top
three areas of concerns?
Source: Wisegate June 2014

The problem is that there is a general acceptance that all three top risks are growing—more
than 80% of participants believe that major risks are increasing in their industry (see Figure
2).
[Note: These three ‘top risks’ are non-specific—they are whatever the participant
considered to be his or her personal top three risks. Overall, the top three risks are
malware, data breaches and outsider threat.]
Figure 2: Survey Question: Which risks are growing for your specific company and
industry?
Source: Wisegate June 2014
What this means, in effect, is that IT cannot accurately communicate an increasing security
risk to Business; and Business cannot accurately understand that security risk and its
possible impact on the business.
Is This Important?
This lack of communication is very important, for three particular reasons:
» Real security cannot be achieved without full Business buy-in.
» Business is likely to become suddenly very keen on understanding security
following the recent prosecution of FedEx in what can be seen as an extension of
the ‘failure to prevent’ theory. “This bodes ill not only for corporations that fail to
prevent criminal activity, but for corporate compliance officers whose programs,

when scrutinized under the glare of 20-20 hindsight, may be found deficient.”1
It is
possible that within a relatively short period, individual board members could be
held legally liable for security failures.
» Boards are being urged by the National Association of Corporate Directors to be
more proactive in information security.
The reality is that possibly for the first time, corporate boardrooms are taking cyber security
seriously. The continuous flow of news of major security breaches in major companies is
having an effect. Boards are asking:
» How does our security stack up?
» How do we compare with other companies in our sector?
Without adequate security metrics to answer those questions in the language that Business
understands, IT/Security will miss a major opportunity.
‘Communication is What the Receiver Does’
It is a tenet of communication that you have to listen. There are signs that Business is ready
to listen.
In July 2014 the National Association of Corporate Directors published a new handbook for
its members: Cyber-Risk Oversight2
. Its advice to directors is organized around five key
principles:
1. Directors need to understand and approach cyber security as an enterprise-wide
risk management issue, not just an IT issue.
2. Directors should understand the legal implications of cyber-risks as they relate to
their company's specific circumstances.
3. Boards should have adequate access to cyber security expertise, and discussions
about cyber-risk management should be given regular and adequate time on the
board meeting agenda.
4. Directors should set the expectation that management will establish an enterprise-
wide, cyber-risk management framework with adequate staffing and budget.
5. Discussion of cyber-risks between boards and senior managers should include
identification of which risks to avoid, accept, mitigate or transfer through insurance
as well as specific plans associated with each approach.
1
The Rise of 'Failure to Prevent' Crimes and CCO Liability; New York Law Journal (27 October 2014):
http://newyorklawjournal.com/id=1202674374593
2
Cyber-Risk Oversight Handbook (free to NACD members): http://www.nacdonline.org/cyber

That last point highlights the need for discussion between IT/Security and the board. When
the handbook was first published, Internet Security Alliance President Larry Clinton
commented, "Most business leaders do not spend a lot of time talking about ISO standards
and NIST framework. They talk about things like profitability, growth, innovation product
development, price-to-earnings ratios. This publication, perhaps for the first time, attempts
to put cybersecurity squarely within that business context."
But while Business might be ready to listen, there remains a difficulty for IT/Security to
speak in a language that it understands.
What IT/Security is Doing
IT/Security is taking a risk-based approach to defending systems; but it currently lacks the
means to report the risk status to boards and internal business partners.
“CISOs are measuring tactical things,” explains the Assessing IT Security Risks lead author,
Bill Burns. “What metrics exist are events-driven: how much classified data was blocked
from leaving the system; how many malware hits were stopped at the firewall or by the AV
software. But there exists a huge disconnect between such activity-based metrics and
rolling them up into ‘what is the impact of our security programs on the business’.”
The problem, he suggests, is that there remains a tool-centric rather than risk-centric view
of security—and the tools that are available rarely provide metrics that can be combined
into an overall metrics-based company risk report suitable for delivery to the board. This
leads to a failure of communication between IT/Security and Business—which is, says
Burns, a major challenge for IT/Security.
To a large degree this basic problem is a natural result of the security product market,
which comprises a wide range of distinct point products. The natural desire to use a ‘best
of breeds’ approach (that is, to use the best available solution for each separate risk)
doesn’t lend itself to seamless security metrics. The extent of the problem can be seen in
Figures 3 to 6, taken from the survey. The diversity of different products expected to be
used in the next 3-5 years makes seamless and cohesive reporting across the whole
security discipline difficult to achieve—and almost impossible in a format suitable to
present to business management. This is unlikely to change within the next five years.

Figure 3: Survey Question: Which endpoint-targeted security controls will be a top-
priority to you in the next 3-5 years (multiple selections allowed).
Source: Wisegate, June 2014
Figure 4: Survey Question: Which mobility / IoT security control will be most
important to your company in the next 3-5 years?

Figure 5: Survey Question: Which of these Messaging, File/Doc Sharing controls
will be a top priority to you in the next 3-5 years (multiple selections allowed).
Figure 6: Survey Question: Stack-rank these Infrastructure controls by which will be
a top priority to you in the next 3-5 years.

This volume of different products makes communicating strengths and weaknesses in the
corporate security profile in relation to business impact a difficult proposition. “Although this
sounds harsh,” comments Burns, “it results in a failure of the security teams to
communicate in business terms, and for business people to understand security. There’s a
business gap—and it’s one of the biggest challenges I see for Security.”
The Danger in Poor Communication
The two primary dangers of poor communications are:
» A continuing disconnect between Business and Security, leading to underfunding
and weak policy implementation
» A Business concentration on the one set of industry-wide metrics already available:
compliance checklists
Many security teams already believe they suffer from the first, and many more will
increasingly come up against the latter.
“I think we are finally at the point, with so many large scale breaches,” explains Burns, “that
Business is taking Security seriously. Boards are ready to listen if we can learn their
language to speak to them. What they want to know is, ‘are we doing everything we should
be doing; and are we doing what our peers are doing?’”
It is that latter point that leads Business to concentrate on compliance-based security. If the
only metrics available are the compliance regulations, then conforming strictly to those
requirements serves two purposes: firstly it provides a defense against any possible ‘failure
to prevent’ legal challenges; and secondly it provides a likely ‘peer comparison’ point.
Most security professionals do not believe that conforming to a compliance checklist
provides the best possible security. However, unless Security can develop its own metrics
and reporting, Business will inevitably increasingly rely on compliance instead—possibly to
detriment of real security.
What is IT/Security Doing About this Lack of
Communication?
IT/Security readily acknowledges that communication is a problem. “People accept that this
is a problem, and talk about it,” comments Burns. “But not one of the survey participants
could say, ‘I cracked the nut—this is what you have to do to communicate successfully.’”

It is a subject that frequently occurs in Wisegate roundtable discussions. For example, in a
recent Wisegate Live Research call, one CISO with a large financial firm noted:
“The higher you go, the more you need to be able to talk about business drivers in
business language that business can understand. The thing that works best seems
to be stories and analogies—they seem to be the best way to share information with
the more senior individuals in your business.”
—“What are the soft skills required for a career in IT and security?” Roundtable
Talking, however, is not reporting, and stories are not metrics. The reality is that IT/Security
mostly does little more than talk about the problem of metrics and reporting.
What Should IT/Security Be Doing?
The survey shows that IT/Security suffers from a lack of adequate metrics. This translates
into poor communication between IT/Security and Business. In the short term this can be
improved by IT/Security aggregating security point solutions to provide a seamless holistic
risk rating; and then creating the metrics to demonstrate the impact of security on business.
In the longer term, the problem provides an opportunity for security users and security
vendors. As the move towards the adoption of security as a service (SaaS) solutions
gathers pace, security teams can start to insist on the provision of usable metrics as part of
the partner agreement.

PHONE 512.763.0555
EMAIL info@wisegateit.com
www.wisegateit.com
Would you like to join us? Go to wisegateit.com/request-invite/ to learn more and to
submit your request for membership.

Metrics & Reporting - A Failure in Communication

More Related Content

What's hot (18)

Viewers also liked (9)

Similar to Metrics & Reporting - A Failure in Communication (20)

More from Chris Ross (7)

Recently uploaded (20)

Metrics & Reporting - A Failure in Communication