Big Data Analytics to Enhance Security

Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Big Data Analytics to
Enhance Security
Anapat Pipatkitibodee
Technical Manager
anapat.p@Stelligence.com

Agenda
• Big Data Analytics
• Security Trends
• Example Security Attacks
• Integrated Security Analytics with Open Source
• How to Apply ?

Big Data Analytics

Everyone is Claiming Big Data

Traditional vs Big Data

Drivers of Big Data
• About 80% of the world’s data are semi-structured or unstructured.

Open Source Tools in Big Data
• Hadoop ecosystem
• NoSQL database

Apache Hadoop Stack
Reference:
Hadoop Essentials
by Swizec Teller

https://whatsthebigdata.com/2016/02/08/big-data-landscape-2016/

Big Data Analytics
• The process of examining large data sets
containing a variety of data types i.e., big
data.
• Big Data analytics enables organizations to
analyze a mix of structured, semi-structured,
and unstructured data in search of valuable
information and insights.

Security Trends

Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection
systems
• 2nd generation: Security information
and event management (SIEM)

Limitation of Traditional SIEMs
Storing and retaining a large quantity of data was not economically feasible.
Normalization & datastore schema reduces data
Traditional tools did not leverage Big Data technologies.
Closed platform with limited customization & integration options

Security Trend from Y2015 to Y2016
Fireeye M-Trends Report 2016

Security Trend from Y2015 to Y2016
• Threats are hard to investigate
Fireeye M-Trends Report 2016

All Data is Security Relevant = Big Data
Servers
Storage
DesktopsEmail Web
Transaction
Records
Network
Flows
DHCP/ DNS
Hypervisor
Custom
Apps
Physical
Access
Badges
Threat
Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-
Malware
Vulnerability
Scans
Traditional
Authentication

Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection
systems
• 2nd generation: Security information
and event management (SIEM)
• 3rd generation: Big Data analytics in
security (Next generation SIEM)

Example Security Attacks

Advanced Persistent Threats
• Advanced
– The attack can cope with traditional security solutions
– In many cases is based on Zero-day vulnerabilities
• Persistent
– Attack has a specific goal
– Remain on the system as long as the attack goal is not met.
• Threat
– Collect and steal information-Confidentiality.
– Make the victim's system unavailable-Availability.
– Modify the victim's system data-Integrity.

Example of Advanced Threat Activities
HTTP (web) session to
command & control
server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exeCalc.exe
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in .pdf,
Emails
to the target
MAIL
Read email, open attachment
Threat intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security

Link Events Together
Threat intelligence
Auth - User Roles,
Corp Context
Host
Activity/Security
Network
Activity/Security
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
MAIL
.pdf Svchost.exeCalc.exe
Events that
contain link to file
Proxy log
C2 communication
to blacklist
How was
process started?
What created the
program/process?
Process making
C2 traffic
Web
Portal.pdf

Correlated Security Log
Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer name:
ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and Settings/smithe/Local
Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time: 2009-01-23 03:19:12,Inserted:
2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My CompanyACME Remote,Server:
acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20
Aug 08 08:26:54 snort.acmetech.com {TCP} 10.11.36.20:5072 -> 10.11.36.26:443 itsec snort[18774]: [1:100000:3]
[Classification: Potential Corporate Privacy Violation] Credit Card Number Detected in Clear Text [Priority: 2]:
20130806041221.000000Caption=ACME-2975EBAdministrator Description=Built-in account for administering the
computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20 TrueName=Administrator
SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1 Status=Degradedwmi_ type=UserAccounts
Sources
All three occurring within a 24-hour period
Source IP
Data Loss
Default Admin Account
Malware Found
Time Range
Intrusion
Detection
Endpoint
Security
Windows
Authentication
Source IP
Source IP

Incident Analysis & Investigation
Search historically - back in time Watch for new evidence
Related
evidence
from other
security
devices

Integrated Security Analytics with
Open Source

SQRRL Solution
https://sqrrl.com/

Anomaly detection in Visualizing
https://sqrrl.com/

Prelert Behavioral Analytics
for the Elastic Stack
http://info.prelert.com/

How to Apply ?

Determining Data That Can Be Collected
Threat
intelligence
Auth - User
Roles
Service
Host
Network
Network Security Through Data Analysisby Michael S
CollinsPublished by O'Reilly Media, Inc., 2014
• Third-party Threat Intel
• Open source blacklist
• Internal threat intelligence
• Firewall
• IDS / IPS
• Web Proxy
• Vulnerability scanners
• VPNs
• Netflow
• TCP Collector
• OS logs
• Patching
• File Integrity
• Endpoint (AV/IPS/FW)
• Malware detection
• Logins, Logouts log
• Active Directory
• LDAP
• AAA, SSO
• Application logs
• Audit log
• Service / Process

Option 1 : Replace All Solution
• Data sent to new Big Data
Analytic Platform
• Big Data Analytic Platform
– Static Visualizations /
Reports
– Threat detection, alerts,
workflow, compliance
– Incident
investigations/forensics
– Non-security use cases
Big Data Analytic Platform
Raw data
Alerts
Static
Visualizations
Forensics / Search
Interface

Option 2 : Big Data to Traditional SIEM
• Data sent to both system
• Big Data Analytic Platform
– Incident
investigations/forensics
– Non-security use cases
• Traditional SIEM
– Static Visualizations /
Reports
– Threat detection, alerts,
workflow, compliance
Big Data Analytic
Platform
Raw data
Forensics / Search
Interface
SIEM
Alerts
Static
Visualizations
Connectors

Factors for evaluating
Big Data Security Analytics Platforms
Factors for Evaluating Open Source
• Scalable data ingestion HDFS
• Unified data management platform Cassandra / Accumulo
• Support for multiple data types Ready to Customized
• Real time Spark / Strom
• Security analytic tools No
• Compliance reporting No
• Easy to deploy and manage Manage many 3rd Party
• Flexible search, report and create new
correlation rule
No

Big Data Analytics to Enhance Security

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Big Data Analytics to Enhance Security (20)

More from Data Science Thailand (20)

Recently uploaded (20)

Big Data Analytics to Enhance Security