Apply big data and data lake for processing security data collections

Apply Big Data and Data Lake for
processing security data collections
Date: 04.02.2017
Gregory Shlyuger, Ph.D.
Enterprise Technology Architect, Mount Sinai PPS
SPIE Presentation (2017)

Agenda 2
» Cyber Security – Modern Enterprise Thread
» SIEM Conceptual Architecture
» SIEM Implementation – What can go wrong?
» Data Lake / SIEM Integrations
» Apache Metron, Security Data Analytics Platform – Next SIEM Evolution
» Use Case - Adding Squid Proxy Logs to Metron Platform

Cyber Security – Modern Enterprise Thread
62%
Increase in Cyber Security
Breaches since 2013.
3
More than 200 days
Average time an advanced
security breach goes unnoticed.
More than 3 Trillion
Total cost of cyber Security
breaches.
1 in 3
Security professionals are not
familiar with cyber security
threads.

4
In 2005 Mark Nicolett and Amrit
Williams from Gartner introduced
term “Security Information Event
Management” (SIEM).
SIEM = SIM + SEM

5SIEM Conceptual Architecture
Vulnerability Scans
User Information
Asset Information
Threat Intelligence
Contextual Data
Operating Systems
Applications
Devices
Databases
Event Data
SIEM
System Outputs
Analysis, Reports, Real Time Monitoring
System Inputs
Data Connection
Normalization
Aggregation
Correlation
Logic Rules

SIEM Components
1. Data Aggregation:
Log management aggregates data from many sources, including network, security, servers,
databases, applications.
2. Correlation:
Looks for common attributes and links events together into meaningful bundles. This
provides the ability to perform a variety of correlation techniques to integrate different
sources, in order to turn data into useful information.
6

SIEM Components
3. Alerting
The automated analysis of correlated events and production of alerts, to notify recipients of
immediate issues. Alerting can be sent to a dashboard or sent via third party channels such
as email.
4. Dashboards
Tools can take event data and turn it into informational charts to assist in seeing patterns, or
identifying activity that is not forming a standard pattern.
7

SIEM Components
5. Compliance
Applications can be employed to automate the gathering of compliance data, producing
reports that adapt to existing security, governance and auditing processes.
8

SIEM Components
6. Retention
employing long-term storage of historical data to facilitate correlation of data over time, and
to provide the retention necessary for compliance requirements. Long term log data
retention is critical in forensic investigations as it is unlikely that discovery of a network
breach will be at the time of the breach occurring.
7. Forensic analysis
The ability to search across logs on different nodes and time periods based on specific
criteria. This mitigates having to aggregate log information in your head or having to search
through thousands and thousands of logs.
9

SIEM – What Can Be Wrong With Implementation 10
SIEM
1.
Collect
Everything
2.
Poor
Source
Data Health
3.
Over
Complicate
Network
Models
4.
Too Much
Focus on
top 10
5.
Lost in
Compliance
6.
Using a
SIEM as a
log search
tool
1. Collect Everything: Collect with Specific Plan. Grow your capabilities
methodically and according with your plan.
2. Poor Source Data Health – Ensure signature are up-to-date and
configure the way they should be, and timestamp is correct.
3. Overcomplicated Network Model – Start with a simple, high-level
model. Don’t start with thousands of zones. What Is business requirements?
4. Too Much Focus On Top 10 Event – When looking for a bad guy
looking for destruction. When trying to find attacks, you’ll probably never
see in top 10 lists. Bottom 10 list more interesting.
5. Lost In Compliance – Don’t use off shell compliance. The off-the-shell
solution most likely will require customization.
6. Log Search Tool – Don’t chasing events in logs, build/use automatically
monitor for incidents.

Data Lake As SIEM Enhancement
Data Lake IS NOT Replacement for SEIM
SIEM
• Originated from needs to consolidate Security Data.
• SIEM incapable of scaling to loads of IT Big Volume Data.
Data Lake
• Central location where all security data is Collected and Stored.
• Running on commodity hardware.
• Allow effectively applying Machine Learning and Map Reduce.
11

SIEM / Data Lake Integration – Approach 1.
Data source duplicates the
stream to both a SIEM
connector and the Data Lake.
12

Proc
 Easy deployment through a change of
source configuration.
 Data in the data lake is independent of
SIEM, no downstream implications.
 Raw data is preserved.
 Fairly nonintrusive for the infrastructure.
Only source configuration needs to be
changed.
13
Cons
 Data source needs a way to split
data to two destinations.
 Parsing has to be done separately
in the data lake.
 Data in SIEM cannot be linked to
its raw data in the data lake.

Data is sent to a SIEM
connector, which splits the
data to the SIEM and the
Data Lake.
14

Proc
 Data is already parsed when it
gets to the data lake.
 Data in SIEM can be linked to raw
data in the data lake.
15
Cons
 Connector needs a way to split data to
two destinations.
 Need a connector for all data sources.
 SIEM and data lake get the same data.
 To keep raw data, connector needs a
way to forward data in raw format.
 Missing or wrong parsers result in "lost"
data.

Data is first sent into
the Data Lake and
then forwarded via a
SIEM connector to the
SIEM.
16

Proc
 Filtering can be applied to reduce
the load on the SIEM.
 One stream of data consumes
less bandwidth.
 Data in the Data Lake can be
parsed at any time, and parsing
can be updated.
17
Cons
 SIEM connector needs to support data
formats when reading from the Data
Lake.
 Data in the Data Lake needs to be
parsed separately.

Data is picked up by
the SIEM first and then
forwarded on to the
Data Lake.
18

Proc
 All data from the SIEM (including
alerts) can be forwarded to the
Data Lake.
 Parsed data is available in the
Data Lake.
 Existing environment can be
upgraded easily without much
change to the existing setup.
19
Cons
 SIEM needs a way to export the data to a Data
Lake.
 SIEM stays the bottleneck for performance.
 Needs a connector for all data sources.
 SIEM and Data Lake get the same data. No
pre-filtering for SIEM.
 Raw data is hard to preserve.
 Missing or wrong parsers result in "lost" data.

Security Data Analytics
Platform
20Apache Metron – Next SIEM Evolution
2013 - Project Started By Cisco.
2015 - Accepted Into Apache Incubation.
2016 - Apache Metron v 0.1 was release.
2017 - Apache Metron v 0.3.1 was release.

21Use Case – Adding Squid Proxy Log To Metron Platform
ImplementationWhat Is Squid?
Squid is a caching proxy for the Web supporting HTTP, HTTPS,
FTP, and more.
It reduces bandwidth and improves response times by caching and
reusing frequently-requested web pages.
Business Requirements:
Need to add proxy events from Squid logs in real-time to existing
real time security monitoring.

22Use Case – Adding Squid Proxy Log To Metron Platform
Platformmplementation1. Proxy event needs to be enriched so that
the domain names are enriched with the IP.
2. In real-time, the IP within the proxy event
must be checked for threat intel feeds.
3. If there is a threat intel hit, an alert needs to
be raised.
4. The system should provide the ability to
configure rules and prioritize/score different
types of alerts.
The end user must be able to see the
new telemetry events completely
enriched from the new data source.
User should be able to see the alerts
prioritized by the high priority with the
corresponding data.
Be able to deploy a machine learning
model that derives additional insights
from the stream.
*All of these requirements will need to be implemented without writing any new code.
5.
6.
7.

23Implementation Use Case on Apache Metron

24Real-Time Enrichment Telemetry Events - BEFORE
24
When you make an outbound http connection to
https://partner.mountsinai.org from a given host, the following entry
is added to a Squid file called access.log.
4861576382.3812 161 387.8.445.068 TCP_MISS/200 107501 GET
https://partner.mountsinai.org – DIRECT/199.27.74.04 text/html
The domain name of the outbound connection.
Unix Epoch Time. IP of host where connection was made.

25Real-Time Enrichment Telemetry Events - AFTER
25
Magic that Metron will do - telemetry event as it is streamed
through the platform in real-time will be processed.
Convert from Unix Epoch
to Time Stamp.
4861576382.3812 161 387.8.445.068 TCP_MISS/200 107501 GET
https://partner.mountsinai.org – DIRECT/199.27.74.04 text/html
IP of host where connection was made,
Use Metron’s asset enrichment.
Use the Metron’s Threat Intel Services to cross-reference the IP with threat Intel feed.

Apply big data and data lake for processing security data collections

More Related Content

What's hot (20)

Similar to Apply big data and data lake for processing security data collections (20)

Recently uploaded (20)

Apply big data and data lake for processing security data collections