2 21677 splunk_big_data_futureofsecurity

Splunk, Big Data and
the Future of Security
WH ITE PAPE R

WH ITE PAPE R

The Changing Nature As many businesses realized that a majority of data exfiltration
happened when employees were not happy or left the company,
of Security Threats data loss prevention (DLP) was added. Also during this time, due
to the business scandals of the late ‘90s and early 2000s, access
Current IT security tools and mindsets are no longer controls and system monitoring was mandated and regulatory
compliance became a cost of doing business. Funding shifted
adequate to meet the scope and complexity of
from security to compliance and vendors rushed to create more
today’s threats. Internet security has evolved over the canned reports “chasing the money.”
last ten years but advanced persistent threats and While the SIEM “funnel” or data reduction approach was
the sophistication of the malware have fundamentally helpful in reducing the amount of data the team had to analyze
changed the way security teams must think about to recognize attacks, attackers exploited the fact that the
security team had only a subset of all data to work with due
these new threats and the tools used for detective to data scalability issues. In addition, silos remained between
controls. IT operations, IT security and development teams in large
IT organizations. Applications closest to the core business
product or service mission were monitored by IT operations or
The Evolution of the IT Security Infrastructure
applications teams or in some instances not at all. Developers
Over the years, security teams have moved from simply created applications that either didn’t log or logged in ways
protecting the IT infrastructure perimeter with traditional that were only understood by the developers and that were not
defenses (firewalls), anti-virus and intrusion detection systems standardized. This data was not able to be included in a SIEM
(IDS). They have employed sophisticated IDS to prevent attacks view due to complexity or expense.
and we watched as these evolved into intrusion prevention
Not having access to application data meant that security teams
systems (IPS). Even with this level of protection, it was not
had a blind spot for risks to the business as malware and viruses
possible to track all the events that security teams saw from
that took advantage of undisclosed or zero-day application layer,
the security architecture and many of the IDS events were
the lack of context for security incidents, and silos between
false-positives. Security event management systems (SEM),
teams that increased incident response times.
which correlate data from these systems, evolved to reduce the
workload using correlation rules to reduce false-positives, alert
on possible threats and provide some visualizations and canned Exploiting Trust
reports that reflect some security metrics.
Today, who we are, what we do, and who we know is available on
LinkedIn, Facebook and other social networking sites. A simple
“…There is now a fairly accepted consensus that the technology search on Google can reveal our PowerPoint presentations and
(AV, IPS, and Firewalls) aiming to keep malicious actors at bay isn’t the conferences we’ve attended. Today attackers target the most
completely successful.” vulnerable point in the network—exploiting human trust using
spear phishing. The exposure to today’s sophisticated advanced
Chris Silva,
persistent threat (APT)-style attacks and the pervasive presence
IANS Trend Report 20111
of malware is a harsh wake up call for many IT organizations.
According to Kevin Mandia of Mandiant Inc., “…there are
thousands of companies compromised—actively, right now,”2
As use of the web and email for business exploded, over
with highly sophisticated malware that has yet to be discovered
time (and in no particular order) web proxy, email security,
deposited by persistent attackers.
vulnerability assessment, database security were added to
monitor possible attacks, understand what systems were An appropriate analogy of this new style of targeted attack
unpatched and vulnerable, and prevent the spread of viruses and would be the difference between a car thief that wanders around
malware on systems. This deluge of data proved too much for parking lots at a variety of locations looking for cars that may
the SEM and the security Information management system (SIM) be unlocked with keys left behind the visor, versus a thief that
was introduced to reduce the workload on the SEM by collecting, wants your specific car. The latter type of thief decides to follow
normalizing and storing data from the security architecture you around for weeks or months, watching your habits, learning
sending a subset of this data to the SEM. where you live and getting to know who you talk to, where you
go and who you know. He then introduces himself to you at
This combination SIM/SEM technology and any required
a party as a co-worker of one of your best friends in another
collection agents became referred to as the security information
department and then asks to borrow your car.
and event management system (SIEM) and continued the data-
reduction or data-distilling strategy first introduced by the SEM. The “data-thief” you linked to on Linkedin wasn’t really a long
This architecture is usually visually represented as the funnel: forgotten coworker but someone you’ve never met who knows
Information is gathered from a wide variety of signature-based you very well. The ‘data-thief’ sends you an email with a PDF
systems and operating systems and a sub-set of the collected attachment called “organizational changes” that contains zero-
data is sent to the SEM and correlated using a rule-based system day exploit code. With one click, your system is ”owned” and
focused on known threats such as classic perimeter-based your email, private information, your website content or other
attacks over open ports or brute force attacks. company intellectual property data is potentially compromised.

listen to your data 2

WH ITE PAPE R

What’s worse is if you happen to have admin rights to your a year ago with new information. In July of 2011, McAfee
system and stored credentials on your computer to other critical announced that it had found active malware deposited on systems
applications on the network. The deposited malware can change in 2006. Re-analyzing five years of log data is outside the reach of
settings on your Windows group policy object and change the traditional SIEM and more suited for a big-data solution.
Windows DLL startup order, guaranteeing that the malware
“Most security software prevents or detects a high number of
will start at boot time and stay persistent. It can also replace or
known threats. While you need to have these capabilities in
switch on a little used networking service to spread the malware
order to detect the botnets and viruses that cause interruptions
to other systems. This type of malware may also stay resident
to your organization’s daily operations, they miss the advanced
but dormant on several other systems as a backup in case the
threats being used to target your most sensitive information.
first compromised system is found. Each of these steps may take
Additionally, much of this software — although not all of it — is
place several days or weeks apart.
designed to limit your control over what threats are detected,
While the threat landscape has changed, the mindset of the how the detection occurs and when you remediate.”5
security team has remained focused on a conventional approach
When the attacker sees that one of their “owned” hosts is no
to security dealing with known threats, which includes:
longer available, the attacker does their own post mortem on the
Protecting all information assets discovered host and shifts tactics to get more firmly embedded
on hosts in the network. New command and control instructions
Maintaining controls that are primarily signature-based and
are sent to other dormant malware on hosts on the network and
preventative
the persistent attack continues.
Paying the most attention to the egress points in the
network The Implications of New Modus Operandi
Collecting logs for compliance and post incident forensics
The attacker wants their malware to behave like a normal
Getting smart on the current malware threats application so that log data doesn’t cause any security systems
alarms. If an attacker knows you well and establishes trust with
Finding and removing malware and infections
you over a real or imagined long period of time, then there’s
Preserving the network: a successful outcome defined as a good chance you will follow a request of an attacker telling
no attackers get in. 3 you to check out a website, click on an attachment or give to a
charity that needs help now (and pay with a credit card).
Our Overdependence on SIEM Because of the current mindset, security professionals always
With hundreds of variants to social engineering-based scenarios, find these words hard to admit: completely preventing these
many ways to compromise a system, zero-day vulnerabilities in kinds of attacks is futile. However, in the current environment,
applications and operating systems and thousands of malware the security team can work to minimize the damage by quickly
variants, what are the odds that one of your 200+ SIEM rules will spotting new user or machine patterns of behavior in very large
fire and alert you to a problem? “Traditional security information data sets that may be worth investigating.
event management (SIEM) systems typically don’t detect a
relentless targeted attack designed to avoid raising any red flags: Thinking Like an Attacker to Find
they’re tuned to catch unusual activity, not stealthy attacks that Unknown Threats
hide behind legitimate user credentials or normal traffic.”4
For years security professionals have been in reactive mode
when a security threat was recognized. As a threat was
“‘Can I be compromised?’ is no longer the right question to be recognized, the team was dispatched to deal with what was
asking.” detected. If it was determined that the issue was with a user’s
Jason Rebholz, behavior, the team would try to educate the user about the
Mandiant, 2011 importance of data protection and move on to the next problem.
But for each incident the team was able to address, there were
others that went undetected. The current security tool sets by
If we are lucky, and a piece of our security infrastructure finds a design (including SIEM) have us thinking like a victim and not like
compromised host with persistent malware, we declare victory, an attacker. To understand the full scope of an attack we need to
clean or re-image the affected host and are satisfied with our start thinking like an attacker.
success. Most SIEMs act in a serial fashion—the security event
Thinking like an attacker means you must understand:
detected at the SIEM level is presented to us as the “end-of-the-
story” supported by evidence from signature based systems. Asset and data criticality
With this approach, the SIEM will not tell us that the security
Location of the most important company data assets
event that was alerted on was really compromised host number
117 and that the attack really started with a different host over Ways your systems and data can be accessed
a year ago. The information to support this possibility has long
Means by which malware can be spread in the organization
since passed into the history books. The SIEM is not architected
in a way that will let you easily re-examine an old attack from Means by which malware can be made persistent
Who might be the most “attractive” victims, what level
they are in the enterprise and what data they might have
access to

WH ITE PAPE R

Knowing what would be considered unusual accesses to Dealing with Unknown Threats Using
important data based on time, frequency or location
Big Data and Analytics
Discovery of a single compromised host should not end the
Just as businesses use business intelligence solutions to monitor
security investigation
large amounts of customer data and watch for patterns that
allow them to better understand customer behavior, security
professionals need similar solutions for the infrastructure.
They need to monitor network, host, and application behaviors
in a contextual way across IT data to understand the depth and
breadth of persistent malware in the IT environment.

“There are some emerging use cases for information security which
can only be handled with big data capabilities.”
Neil MacDonald,
Gartner, April 12, 2011

Enter Splunk—The First Big Data System
Mandiant’s “Anatomy of a Hack”
for Security
Splunk Enterprise is the engine for machine data. Splunk
Understanding the attacker means monitoring large data sets of software enables enterprises to gain operational intelligence
normal user activity data looking for patterns of activity that are by monitoring, reporting and analyzing real-time machine data
not normal in context of time, place, or appropriateness. This has as well as terabytes of historical data located on-premise or in
given rise to a new role on the security team called the security the cloud. With Splunk you can leverage an analytics command
intelligence analyst. 6 These individuals: language to map and visualize any potential attack scenario
Take the “actor view” to understand the identities, goals against the business’ most important data assets. These
and methods of potential adversaries scenarios can be easily aligned with the business risk-based
modus operandi of potential attackers. Automated searches can
Work with management, lines of business and operations continuously monitor for abnormal patterns of behavior in host,
personnel—this knowledge makes them aware of the network and application data. Combined with an understanding
threats posed by persistent adversaries of where critical data is stored, who should have access to it,
Assess actions and determine if a pattern of threatening time based analysis of typical user behavior (i.e., how much
behavior is emerging mail is sent per day, normal data access times, physical access
and normal host network behaviors), abnormal patterns can be
Map and visualize threat behavior patterns against big- detected. Adaptive monitoring of the active phase of the attack
data sets of normal IT activities with analytics over time presents opportunities to detect abnormal behaviors
“The core of the most effective [APT] response appears to on hosts and networks.
be a new breed of security analytics that help quickly detect
anomalous patterns— basically power tools in the hands of a
“Splunk has approximately 3,000 customers, the vast majority
new and important sub-category of data scientists: the security
of which are using Splunk to solve big data problems—providing
analytics expert.” 7
operational intelligence to make machine data accessible, usable,
APT style attacks have de-positioned SIEM moving it from a and valuable.
solution to simply a tool for monitoring mainly known threats. Frank Sparacino,
First Analysis, September 15, 2011

“Because adversaries are intelligent, well funded and patient, they
can afford to take weeks to probe their targets and months to plant Specific sets of automated Splunk searches of normal user
malware inside the organizations in order to exfiltrate data.” activities may comprise multiple scenarios. A single search
Chris Silva, can trigger several other searches in a decision tree fashion
IANS Trend Report 2011 8 and could confirm the existence and spread of malware. An
anomalous behavior detected can be analyzed along with other
time-sequenced IT data and changes to host configuration files.
Old security events can and should be reviewed and reanalyzed
on a regular basis as a means of preventing re-infestation and
as a way of determining whether the first compromised victim
found was the only one, the first one or victim 112.
An advanced approach—one that uses big-data and analytics
for tracking and discovering malware left behind by persistent


WH ITE PAPE R

attackers--can move the security team from a conventional Security teams need to start using their creativity to think
approach to a more flexible and advanced approach more about the modus operandi of the attacker and work with the
suitable for the newest security threats. Best practices of this business, assigning risk to data. Thinking like an attacker and
new approach include: modeling attacks that start with spear-phishing against the most
important business assets aligns the security team with business
Focus efforts on most important data business assets objectives through prioritization of data assets and risk. This
Use detective controls linked to data analytics watching for type of thinking is a valued skill.
behavioral outliers Splunk is a security intelligence solution for monitoring large
Seek, model and dissect attack patterns datasets and gives you the ability to tell the difference between
humans interacting with IT systems and behaviors that may
Develop deep understanding of attackers’ modus be caused by malware. Splunk can cover known threats via
operandi in context of the organization’s key assets and IT information from signature and rule-based systems but also can
environments be used to monitor for unknown threats based on risk-based
Realize that attackers will sometimes get in, but are scenarios translated into Splunk’s analytics language.
detected quickly and impact (risk) is minimized
The security team’s creativity and imagination are supported
through ad-hoc exploration of data and modeling of attacks Free Download
based on business risk. This allows security teams to speculate Download Splunk for free. You’ll get a Splunk Enterprise
on potential attack vectors in advance of any actual attack. license for 60 days and you can index up to 500 megabytes
Different critical alerts can be created to support a search that of data per day. After 60 days, or anytime before then, you
has found a particular issue on one host versus another. can convert to a perpetual Free license or purchase an
Enterprise license by contacting sales@splunk.com.
Dealing with Known Threats
While the focus of this paper is a discussion of a new paradigm
and a new way of thinking about detecting unknown threats, it is
not meant to be a condemnation of practices around monitoring
known threats. Script-kiddies and canned attack tools are still
out there. There is a still a need to monitor security operational
metrics for continuous improvement. System patching, IPS
attacks, firewall accepts/denies, DNS logs, data loss prevention
(DLP) systems, anti-virus and other endpoint security systems
should still be monitored.
Splunk offers alternatives to users who would typically purchase
a SIEM to monitor their security infrastructure only monitoring
known threats. Splunk (as of this writing) offers over 30 security
apps free of charge that can monitor specific security point
problems. Splunk also offers the Splunk App for Enterprise
Security, which monitors over 100 security metrics, provides
over 160 reports, identity correlation and a complete set of the
most important correlation searches to offer SIEM functionality.
It includes incident workflows and supports drill-down into raw
data as well as workflow actions to launch cross data-type views 1 IANS Trends: A Behavioral Approach To Threat Modeling, 2011
of incident data. Just as with the core Splunk product, real-time
2 Report Details Hacks Targeting Google, Others, WIRED, Kim Zetter, February 3, 2010
alerts can be generated.
3 When Advanced Persistent Threats go Mainstream, Security for
Business Innovation Council, July 11, 2011
Summary–Reaching for Security Intelligence
4 APT Shaping SIEM, Kelly Higgins, Security Dark Reading, 10/3/2011
Finding anomalous patterns in massive data sets over time and
in context for unknown threats is the key to detecting advanced 5 MANDIANT M-Trends Report 2011, Mandiant Inc.

persistent attackers and the malware they leave behind. SIEMs 6 IANS Trends: A Behavioral Approach To Threat Modeling, 2011
that are set up to monitor security infrastructure watching for
known threats do not solve the APT problem and have security 7 When Big Data Met Security: Is The New Era Beginning? Chuck Hollis, VP – CTO, EMC
Corporation, April 12, 2011 http://chucksblog.emc.com/chucks_blog/2011/08/when-big-data-
teams in constant cleanup mode thinking like the victim and not met-security-is-the-new-era-beginning.html
like the attacker. Only big-data solutions with strong analytics
8 IANS Trends: A Behavioral Approach To Threat Modeling, 2011
and visualization capabilities can provide insight into anomalous
behavior.

250 Brannan St, San Francisco, CA, 94107 info@splunk.com | sales@splunk.com 866-438-7758 | 415-848-8400 www.splunkbase.com

listen to your data
www.splunk.com
Copyright © 2011 Splunk Inc. All rights reserved. Splunk Enterprise is protected by U.S. and international copyright and intellectual property laws.
Splunk is a registered trademark or trademark of Splunk Inc. in the United States and/or other jurisdictions. All other marks and names mentioned
herein may be trademarks of their respective companies. Item # WP-Splunk-Big Data & The Future of Security-102

2 21677 splunk_big_data_futureofsecurity

More Related Content

What's hot (18)

Viewers also liked (8)

Similar to 2 21677 splunk_big_data_futureofsecurity (20)

More from Svetlana Belyaeva (10)

2 21677 splunk_big_data_futureofsecurity