Integrating Cyber Security Alerts into the Operator Display

{
Integrating Cyber Security
Alerts into the Operator Display
Digital Bond, Inc.
Michael Toecker, PE
ddddddddd
EnergySec
9th Annual Security Summit

Ñ  Michael Toecker
Who Am I?

Monitoring and Response of Cyber
Security Events Originating from the
Control System Parallels the Monitoring
and Response of Process Events
The Premise

{
Ñ  ICS Operations was similar to Security
Operations
Ó  ICS had alarms, SecOps had alarms
Ó  ICS had events, SecOps had events
Ó  ICS had historical points, SecOps had
voluminous logs
Ó  ICS had 24/7 Operators, SecOps had
Analysts (some 24/7, others not)
Ó  ICS had a responsibility for monitoring
safe and eﬀective productions, SecOps had
responsibility for ensure secure and
trusted operations
I Spent a Year
working as a
Security Guy
in an
Operations
Environment
ICS Ops vs. SecOps

{
Task
SecOps
ICSOps
Visualizing Data using Graphs,
Charts, etc
X
X
Providing Status Indicators when
parameters went out of normal
X
X
Directed Field Personnel to Take
Specific Actions based on Events or
Alarms
X
X
Reviewing of Logs, Records, and
Other Data to Improve Efficiency
and Locate Problem Areas
X
X
Investigate for Compliance and
Effect on Process, and find ways to
Prevent, Detect and Respond
X
X
What I Often
Saw in ICS
Operations was
Paralleled in
What I was
Doing
Parallels

{
Ñ  …was the data.
Ó  My data was security logs, their data
from process points.
Ó  But we were both identifying
conditions that could impact our
production or compliance, and taking
some action to correct
I was an
Engineer with
Specialized
Knowledge of
Speciﬁc
Equipment
What was Diﬀerent…

{
Ñ  There is an emphasis on procedure,
and process when faced with issues
Ñ  Troubleshooting where advanced
knowledge is required is conducted
by those with the knowledge
Ñ  Operators follow known actions that
will return a system to a stable state,
usually developed by process
engineers.
Operators
Monitor &
Respond, but
Do Not Always
Possess
Speciﬁc
Knowledge
The Role of Operators

Why not add some
Security?

Ñ  Lack of Understanding and Confusion about
Computer Security
Ñ  Owner A]itude is that Security has nothing to
Do with Operations
Ñ  Leads to Reduction in Situational Awareness
Ñ  Operators Don’t Know What Actions To Take
The Problems

Ñ  Proper Notiﬁcation Reduces Response Time to
Security Incidents
Ñ  Regulatory Requirements can be Met With
Existing Personnel
Ñ  Alerts and Events directly to 24/7 Personnel
look Awesome as Compensating Controls
The Beneﬁts

{
Cyber Security
Events &
Incidents
Detectable w/
Security
Monitoring
Security Events
Operators
Could Respond
To
Not a
Substitute for a
Focused
Security
Monitoring
Program
The Limitations

{
Monitor,
and
Analyze
Identify
Security
Conditions
Identify
Operational
Events
Develop
Procedures
for Action
Implement
Condition
and
Procedure
Security
Monitoring
Program
Should Feed
into
Conditions for
Operator Alerts
The Role of Security

{
Monitor
Data Points
Identify
Process
Conditions
Identify
Operational
Events
Develop
Procedures
for Action
Implement
Condition
and
Procedure
This looks a lot
like Process
Intelligence
Process, the
only diﬀerence
is the Analysis
and
Knowledge
….wait a minute.

Identify Speciﬁc Clear Cyber Security Events
Determine Events Appropriate for Operator
A]ention
Create Operations Procedures for Actions
Develop a Detection and Presentation Strategy
The Process

{
Part 1
Identify

Operational

Events

{
…Clear
• No Ambiguity
• Straightforward Yes/No
Decision Point
…Derivable
• Sourced Directly from Control
Systems Security Data, not from
Intuition or Analysis
…Actionable
• Speciﬁc Actions can be taken on
receipt of the Event
• Not Dependent on Other
Events, or on Further Analysis
An Operational
Cyber Security
Event Should
Be..
Identify and Deﬁne

{
Ñ  Questions to Ask
Ó  What do my regulations tell me to be
concerned with?
Ó  What do various standards bodies tell
me to be concerned with?
Ó  Do I have speciﬁc policy statements
that suggest alerting, or 24/7 response?
Ó  What Lessons Learned Do I have
related to Cyber Security?
Identify Cyber
Security
Conditions to
Alert On
Identifying Events
This is my polite way of saying
“If You Got Hacked, How Did
it Happen?”

List of
Security
Conditions
Regulations
Require
Monitoring
and Action
Standards
suggest an
Approach
Security
Policy may
Specify
Conditions
Lessons
Learned from
Security
Incidents
Determine Conditions

{
Ñ  CIP-‐‑007 R4 – Malicious Software Prevention
Ó  Paraphrase: ~..shall use anti-‐‑virus software
to detect malware on all Cyber Assets within
the ESP~
Ó  Conclusion: I should alert on anti-‐‑virus
detections
Ñ  CIP-‐‑007 R5 – Monitoring Electronic Access
Ó  Paraphrase: ~monitoring processes shall
detect and alert for a]empts at or actual
unauthorized access~
Ó  Conclusion: A]empts at unauthorized access
include incorrect passwords, alert on that.
Regulations,
such as NERC
CIP, may
provide clues
as to what
events should
be monitored
Regulations
Well, I did say clues…
Source: NERC CIP Standards, V3

{
Ñ  Section 3.2.2 – Signs of an Incident
Ó  ~Too many indicators exist to
exhaustively list them~
Ó  ~Common ones include multiple failed
login a]empts, deviations from
normal network traffic, filenames with
unusual characters..~
Standards can
help as well,
but still are
clues not firm
guidance
Standards
Source: NIST SP-‐‑800-‐‑61

{
Ñ  What I’ve seen in the past:
Ó  ~Addition and Modiﬁcations of Users
shall be conducted through the change
control process~
Ó  ~New Software on Control Systems
requires approval by the Senior
Manager~
Conditions
may exist in
your
corporations IT
Security
Policies
Policy Remarks

{
Ñ  Good Security Comes with
Experience,
Ó  Most Experience Comes from
Failures in Security
Ñ  ….but it doesn’t have to be YOUR
Failures in Security
Ó  Talk, Listen, Learn
Why
Information
Sharing is
Important.
Lessons Learned

{
There are tons
of events
available, but
not all are
relevant or
appropriate for
Operations
Complex, Irrelevant

{
• Start with from general
security conditions
• Trim to Speciﬁc Events
within those categories
Top
Down
• Start with Every Potential
Event that Could Be
Generated
• Trim to Speciﬁc Events from
the Potentials
Bo]om
Up
There are Two
Main Methods
for Identifying
Events
Methods to Identify

{
Ñ  Speciﬁc Classes of Computer Security
Events
Ó  Virus Detection, Failed Logins,
Disallowed Ports, etc
Ó  Good Source of Some Classes – NIST
SP-‐‑800-‐‑53
Ñ  Useful for PC based systems, which
often have a huge amount of capacity
for security
Top Down is
Good For
Systems with
Many Potential
Events
Top Down Approach

{
Cyber Security
Event Class:

Virus
Detection
Top Down Example

{
Ñ  Enumerate the Security Capabilities of
the Device. Examples:
Ó  Provides Speciﬁc Syslog Evidence
Ó  Sets a Point when a Login Threshold
has been reached
Ñ  Useful for Devices, where Capability
is often limited
BoYom Up is
Good for
Systems with
Limited
Capability for
Security
Bo]om Up Approach

{
Review of Manuals and Datasheets can identify
detectable Cyber Security Events
Bo]om Up Example
Source: S&C IntelliRuptor Instruction Sheet 766-‐‑560

{
Ñ  Top Down
Ó  Allows you to set criteria, and then delve
into system to ﬁnd triggers to meet it
Ó  Avoids the complexity of ge]ing into the
weeds of system events
Ó  May miss important conditions due to
avoiding those same weeds
Ñ  Bo]om Up
Ó  Complex, but most Detailed
Ó  Requires analysis of many events that will
likely never make it in front of an operator
There are
advantages and
disadvantages
of each
Approach
Compare and
Contrast

{
Ñ  Windows Based Computers are the
obvious systems to use Top Down
Ó  Event Heavy, Highly Complex
Ó  Events were designed from an
incident response perspective, not
from an alert perspective
Use Top Down
when a system
is highly
capable of
reporting
security events
to narrow your
range
When to Use an
Approach

{
Ñ  Systems like PLCs, Controllers, some
Network Devices have limited
capability to report security status
Ó  Won’t be able to simply deﬁne events,
you’ll have to work with what’s there
Use BoYom Up
when working
with devices
that report on
few security
conditions
When to Use an
Approach

{
Condition
Source
Anti-‐‑Virus Detection
NERC CIP-‐‑007 R4
User Modiﬁed or Added
Security Logs Deleted
Security Logs Full
Excessive Incorrect Login
Use of Removable Media
Good Practice
New Software Installed
IT Policy
Logging Options
Changed
IT Policy
The End Result
of this Analysis
is a List of
Conditions to
Alert On
List of Conditions
Note: This list is far from comprehensive

{
Part 2
Appropriate for

Operators

{
Ñ  Is the Condition a Clear Cyber
Security Event?
Ñ  Is the Condition Derivable directly
from Logs, Alerts, and other
evidence?
Ñ  Is the Condition Actionable by
Operators?
Not Every
Condition is
Appropriate
for Operator
Notiﬁcation
Appropriate for
Operators

{
Condition
Source
User Modiﬁed or Added
Security Logs Full
Good Practice
New Software Installed
IT Policy
Logging Options
Changed
IT Policy
Unclear
Conditions are
Removed from
the List
Is it Clear?

{
Condition
Source
Security Logs Full
Lesson Learned
Remove
Conditions
Incapable of
being Derived
from Evidence,
or Require
Analysis
Is it Derivable?

{
Condition
Detection
Method
Reliability
Anti-‐‑Virus
Detection
Windows Event
Log
Very Reliable, Test Indicates
an event generated on each
detection in SYSTEM log
Security Log
Deleted
Windows Event
Log
Very Reliable, an explicit
event is created on clearing
Excessive
Incorrect Login
Windows Event
Log
Reliable, so long as the
account lockout se]ings in
SECPOL.msc are set correctly
Use of
Removable
Media
May require 3rd
party program.
Not Always Possible without
3rd Party Program
How Reliable
are the
Detection
Methods? Do
they have
potential false
positives?
Reliable and Unreliable
Conditions

{
Remove
Conditions that
an Operator
cannot
Realistically
take Action On
Is it Actionable?
Condition
Source
Anti-‐‑Virus
Detection
Security Logs
Deleted
Security Logs
Full
Excessive
Incorrect Login

{
Why were some of the conditions removed?
An Aside…
Ñ  User Modiﬁed or Added
Condition
Reason for Removal
User Modiﬁed
or Added
Not Clear, as there are legitimate reasons for adding, or
modifying a User and these reasons aren’t apparent
without analysis.
Security Log
Full
Not Actionable, as operators should be doing
maintenance and admin functions.
Removable
Media
Not Derivable, on most systems as is. May require a 3rd
Party program to do a decent job of this.

{
Ñ  Example: Removable Media Detection
Ó  Wasn’t able to do this in Native
Windows in a Clear and Derivable
manner
Ó  Use of Third Party tools can change
this, making it possible to monitor and
alert
A Previously
Rejected
Condition can
become valid
with New
Information or
Technology
When Conditions
Change

{
Ñ  USB Based Infection Lesson Learned
Ó  New USB Showed up in Registry Change
Ó  Auto-‐‑Run Shows up in Registry Change
Ó  Addition of Programs to the “Run” and
“RunOnce” keys in the Registry
Ó  Copying of Files into “System”,
“System32”
Ñ  Is this Clear? Deﬁnable? Actionable?
Some of the
More
Advanced
Conditions
That We Can
Deﬁne
Let’s Get Crazy…

{
List of
Conditions has
been
generated,
what next?
What Comes Next?
Condition
Detection
Method
Reliability
Anti-‐‑Virus
Detection
Windows
Event Log
Very Reliable, Test
Indicates an event
generated on each
detection in SYSTEM log
Event Log
Was Cleared
Windows
Event Log
Very Reliable, an explicit
event is created on
clearing
Excessive
Incorrect
Login
Windows
Event Log
Reliable, so long as the
account lockout se]ings
in SECPOL.msc are set
correctly

{
Part 3
Create Operations

Procedures

{
Ñ  Notifying Operators of Cyber Security
Events is useless if the Operator has
no action to take
Ñ  This guidance typically takes the form
of Operational Procedures
Ñ  Each Event must have an appropriate
action to be taken
This is Now a
Procedure
Exercise
Operator Actions

{
Ñ  “Notify Lead I&C Engineer by
Phone”
Ñ  “Isolate Infected System From
Network by Disconnecting Ethernet”
Ñ  “Call Out via Radio to check if invalid
login is from authorized user”
Be Succinct
and Speciﬁc
Guidelines for Actions

{
Ñ  No IT Administrative Functions
Ñ  No Maintenance Functions
Ñ  Limit the Analysis Necessary
Ñ  …and don’t give them someone else’s
work
Keep the
Guidance
within
Operator’s
Authorized
Abilities
Guidelines for Actions

{
Personnel
Responsible
Trigger
Actions
Documentation
An Operating
Procedure has a
few common
characteristics
Operating Procedures

Example
Operating Procedure
Bring up Example Procedure

{
Ñ  Case in Point – Conﬁcker (MS08-‐‑67)
Ó  Highly Aggressive Worm which
impacts network communication
Ó  Makes use of very reliable exploit in
Server service
Ó  A]empts to brute force accounts
Ó  Spreads over USB and removable
media as well
Some Cyber
Security Events
may Cause
Production
Impacts
Worst Case Scenario

{
Ñ  A Highly Aggressive worm like Conﬁcker
can have production consequences.
Ó  Continuing to operate while this is going
on is risky.
Ó  Who makes the decision to halt
production? Operator? Shift Supervisor?
Plant Manager?
Ñ  Make sure the information gets to those
make the decision.
What guidance
would prepare
an operator for
these Alarms?
Worst Case Scenario

{
Section 5
Present to

Operator

{
Ñ  Most Cited:
Ó  The Alarm Management Handbook
The High-‐‑Performance HMI
Handbook.
Ñ  Wri]en by Bill Holliﬁeld and Paul
Gruhn
Ó  Of Course, Nothing Speciﬁc on
Security
There is
already a lot of
guidance on
development
of Operator
Displays
Operator Displays

{
Ñ  Help Operators Perceive the Important
Security Data
Ñ  Give Operators Data-‐‑in-‐‑Context
Ñ  Help Them Comprehend the Situation in
Terms of the Process
Ñ  Help Predict Future Status by Providing
Trending
Guidelines for
Cyber Security
Displays
Operator Displays
-‐‑ Tough right now… At least without giving access to an SIEM

Cyber Security
Master Display
Anti-‐‑Virus
Status Display
Users Status
Display
Removable
Media Status
Display
Event Log
Status Display
Concept Operator
Display

{
Ñ  Many HMIs can accept SNMP Traps
Ó  Often used for alerting when hosts
stop communicating
Ó  Security tools can feed this, in certain
conditions
Ñ  Security Logs don’t Translate Well
into traditional displays
Ó  How do you ‘trend’ when you have
thousands of event ids?
Summary:
Limited, and
Nowhere Near
Ideal
Integration with
the HMI

More Research at S4
Ñ  Digital Bond’s S4
Conference in Miami
Beach, January 2014
Ñ  Got an Idea?
Ó  Submit a presentation!
Ñ  Details on
DigitalBond.com

Integrating Cyber Security Alerts into the Operator Display

More Related Content

What's hot (19)

Viewers also liked (15)

Similar to Integrating Cyber Security Alerts into the Operator Display (20)

More from EnergySec (20)

Recently uploaded (20)

Integrating Cyber Security Alerts into the Operator Display