![ELK in Security analytics
-- Lionel Faleiro
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-1-320.jpg)
![About Me
• Trainer and Security Analyst at Institute
of Information Technology / Network
Intelligence India
• 4+ years experience in IT
• Conducted Trainings at multiple
corporates
• Part of the DFIR Team at NII
• Key domains – Security Analytics,
Malware Analysis, Log Analysis, Intrusion
Response
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-2-320.jpg)
![Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-3-320.jpg)
![What Big Data..
• IS:
• Store large volumes of
data
• Enables us to run
queries on the data set
• IS NOT:
• Hadoop, Hive, Pig, Yarn
– these are
technologies
• Does not automatically
give you analytical
results
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-4-320.jpg)
![Why use Big Data in Security?
• User-behaviour
analytics
• Fraud Detection
• Log correlation from
additional sources
• Forensic analysis on
large volumes of data
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-5-320.jpg)
![Known SIEM
Issues
• Unable to ingest a lot of log sources
• Cost of storage is high
• Requires more compute power
• Licensing issues
• Monitoring on each endpoint is
problematic
• Current monitoring is static in nature
• Too many alerts
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-6-320.jpg)
![SIEM + ELK = SOC 2.0
• SIEM Functions
• Alerts for standard IT issues
• Rules based correlations
• Standard reporting/queries
• ELK Functions
• Visualize Logs for anomalies
• Ingest logs from multiple sources
with large volume
• Implement Threat-Hunting strategy
• Custom search and querying
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-7-320.jpg)
![This is not
ELK..
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-8-320.jpg)
![What is ELK?
• E is a NoSQL databased that is based on the Lucene search engine
• Stores data in an unstructured way
• Cannot use SQL to query it.
• L is a log pipeline tool that accepts inputs, executes transformations
and outputs the data into various targets
• K is a visualization layer
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-9-320.jpg)
![ELK Overview
• Beats
• Log shippers – Windows events, system status, network traffic
• Elasticsearch
• Data storage, search engine
• Logstash
• Log management component. Ingest, Process, Output
• Kibana
• - Create visualizations and dashboards
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-10-320.jpg)
![ELK Architecture
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-11-320.jpg)
![Elasticsearch
• Based on Apache Lucene
• Open-source search engine library
• Created by Shay Banon
• Extends Lucene to store, index and search
• JSON over HTTP
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-12-320.jpg)
![Logstash
• Integrated Log management framework
• Log collection
• Centralization
• Parsing
• Storage
• Written in Jruby
• Runs in JVM
• Multiple input mechanism
• TCP/UDP
• Files
• Sysog
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-13-320.jpg)
![Logstash: Conf
• Input {}
• Filter {}
• Output {}
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-14-320.jpg)
![Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-15-320.jpg)
![Kibana
• Visualization platform
• Tight integration with Elasticsearch
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-16-320.jpg)
![Beats
• Filebeat
• Metricbeat
• Packetbeat
• Winlogbeat
• Hearbeat
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-17-320.jpg)
![Filebeat
• A lightweight way to
forward and
centralize logs and
files
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-18-320.jpg)
![Metricbeat
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-19-320.jpg)
![Packetbeat
• Packetbeat is a lightweight network packet
analyzer that sends data to Logstash
or Elasticsearch
• It supports many application layer protocols,
from database to key-value stores to HTTP
and low-level protocols
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-20-320.jpg)
![Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-21-320.jpg)
![Winlogbeat
• Winlogbeat live streams Windows event logs to Elasticsearch and
Logstash in a lightweight way
• Read from any windows event log channel
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-22-320.jpg)
![Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-23-320.jpg)
![Heartbeat
• Monitor services for their availability with active probing
• Heartbeat pings via ICMP, TCP, and HTTP, and also has support for TLS,
authentication and proxies.
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-24-320.jpg)
![Use Cases
• Nginx/Apache
• Sysmon Integration
• Forensics Imaging
Lionel Faleiro [ @sandmaxprime ]](https://image.slidesharecdn.com/nullmumbai-elk-171201090015/85/ELK-in-Security-Analytics-25-320.jpg)
ELK in Security Analytics
- 1. ELK in Security analytics -- Lionel Faleiro Lionel Faleiro [ @sandmaxprime ]
- 2. About Me • Trainer and Security Analyst at Institute of Information Technology / Network Intelligence India • 4+ years experience in IT • Conducted Trainings at multiple corporates • Part of the DFIR Team at NII • Key domains – Security Analytics, Malware Analysis, Log Analysis, Intrusion Response Lionel Faleiro [ @sandmaxprime ]
- 3. Lionel Faleiro [ @sandmaxprime ]
- 4. What Big Data.. • IS: • Store large volumes of data • Enables us to run queries on the data set • IS NOT: • Hadoop, Hive, Pig, Yarn – these are technologies • Does not automatically give you analytical results Lionel Faleiro [ @sandmaxprime ]
- 5. Why use Big Data in Security? • User-behaviour analytics • Fraud Detection • Log correlation from additional sources • Forensic analysis on large volumes of data Lionel Faleiro [ @sandmaxprime ]
- 6. Known SIEM Issues • Unable to ingest a lot of log sources • Cost of storage is high • Requires more compute power • Licensing issues • Monitoring on each endpoint is problematic • Current monitoring is static in nature • Too many alerts Lionel Faleiro [ @sandmaxprime ]
- 7. SIEM + ELK = SOC 2.0 • SIEM Functions • Alerts for standard IT issues • Rules based correlations • Standard reporting/queries • ELK Functions • Visualize Logs for anomalies • Ingest logs from multiple sources with large volume • Implement Threat-Hunting strategy • Custom search and querying Lionel Faleiro [ @sandmaxprime ]
- 8. This is not ELK.. Lionel Faleiro [ @sandmaxprime ]
- 9. What is ELK? • E is a NoSQL databased that is based on the Lucene search engine • Stores data in an unstructured way • Cannot use SQL to query it. • L is a log pipeline tool that accepts inputs, executes transformations and outputs the data into various targets • K is a visualization layer Lionel Faleiro [ @sandmaxprime ]
- 10. ELK Overview • Beats • Log shippers – Windows events, system status, network traffic • Elasticsearch • Data storage, search engine • Logstash • Log management component. Ingest, Process, Output • Kibana • - Create visualizations and dashboards Lionel Faleiro [ @sandmaxprime ]
- 11. ELK Architecture Lionel Faleiro [ @sandmaxprime ]
- 12. Elasticsearch • Based on Apache Lucene • Open-source search engine library • Created by Shay Banon • Extends Lucene to store, index and search • JSON over HTTP Lionel Faleiro [ @sandmaxprime ]
- 13. Logstash • Integrated Log management framework • Log collection • Centralization • Parsing • Storage • Written in Jruby • Runs in JVM • Multiple input mechanism • TCP/UDP • Files • Sysog Lionel Faleiro [ @sandmaxprime ]
- 14. Logstash: Conf • Input {} • Filter {} • Output {} Lionel Faleiro [ @sandmaxprime ]
- 15. Lionel Faleiro [ @sandmaxprime ]
- 16. Kibana • Visualization platform • Tight integration with Elasticsearch Lionel Faleiro [ @sandmaxprime ]
- 17. Beats • Filebeat • Metricbeat • Packetbeat • Winlogbeat • Hearbeat Lionel Faleiro [ @sandmaxprime ]
- 18. Filebeat • A lightweight way to forward and centralize logs and files Lionel Faleiro [ @sandmaxprime ]
- 19. Metricbeat Lionel Faleiro [ @sandmaxprime ]
- 20. Packetbeat • Packetbeat is a lightweight network packet analyzer that sends data to Logstash or Elasticsearch • It supports many application layer protocols, from database to key-value stores to HTTP and low-level protocols Lionel Faleiro [ @sandmaxprime ]
- 21. Lionel Faleiro [ @sandmaxprime ]
- 22. Winlogbeat • Winlogbeat live streams Windows event logs to Elasticsearch and Logstash in a lightweight way • Read from any windows event log channel Lionel Faleiro [ @sandmaxprime ]
- 23. Lionel Faleiro [ @sandmaxprime ]
- 24. Heartbeat • Monitor services for their availability with active probing • Heartbeat pings via ICMP, TCP, and HTTP, and also has support for TLS, authentication and proxies. Lionel Faleiro [ @sandmaxprime ]
- 25. Use Cases • Nginx/Apache • Sysmon Integration • Forensics Imaging Lionel Faleiro [ @sandmaxprime ]