SlideShare a Scribd company logo
AWS Meetup
Chicago
Who am I
Asaf Yigal
Co-Founder and VP Product @logz.io
Email: asaf@logz.io
Twitter @asafyigal
Agenda
• Why do we need Log analytics?
• Intro to ELK
• What is Logz.io
• Installing ELK on your own
• Our Architecture
• EC2 machine comparison
Why do we need Log analytics?
Werner
Vogels
AWS CTO
“Log Analytics is
Fundamental for
Building Cloud
Applications”
Product
Management
Business
Analysis
Customer
Success
BI
Monitoring
DevOps
IoT
Troubleshooting
Support
QA
IT OPPS , ITOA
Compliance
SecOps
SIEM
Multiple Use-
Cases
Log driven development
• Errors, Warnings and exceptions
• Metrics
• Alerts
• Dashboard
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Why
Open
Source
*based on Logz.io research
The Market is Dominated by
Open Source Solutions
Over the past 3 years, the market shifted attention
from proprietary to open source
ELK Stack,
400,000+
companies
Splunk, Sumo Logic,
Loggly, - 20,000
companies
Graphite has > 1M
companies using it
ELK Popularity
Intro to ELK
Logstash
•Streaming data digestion
•Time normalization
•Field extraction
Elasticsearch
•Schema-less search DB
•Highly scalable
Kibana
•Visualization
Open source ELK +/-
Simple and
beautifulIt’s simple to get started and play with ELK
and the UI is just beautiful
Open Source
The largest user base with a vibrant open
source community that supports and
improves the product
Fast. Very fast.
Built on the Elasticsearch search engine, ELK
provide blazing quick responses even when
searching through millions of documents
Hard to Scale
Data piles up and organization experience
usage bursts. It’s super-complex building
elastic ELK deployments that can scale up and
down
Poor Security
Logs include sensitive data and open source
ELK offers no real security solution, from
authentication to role based access
Not Production Ready
Building production ready ELK deployment is a great
challenge organization face. With hundreds of different
configurations and support matrix, making sure it’s always
up is difficult
Up and running in
minutesSign up in and get insights into your
data in minutes
Logz.io Enterprise ELK Cloud
Service
Production ready
Predefined and community designed
dashboard, visualization and alerts
are all bundled and ready to provide
insights
Infinitely scalable
Ship as much data as you want
whenever you want
Alerts
Unique Alerts system proprietary built on
top of open source ELK transform the ELK
into a proactive system
Highly Available
Data and entire data ingestion
pipeline can sustain downtime in full
datacenter without losing data or
service
Advanced Security
360 degrees security with role based
access and multi-layer security
Installing ELK on your own
Prototype
• Installing ELK stack on a single server – 1hr
• Shipping one type of log – 1hr
• Log parsing – 2 hr
• Building Kibana Dashboard – 2hr
• 6 hours to get a simple Prototype
Turning ELK Production ready
OS Level
OptimizationElasticsearch require a lot of OS level
optimization in order to run properly.
Elasticsearch
Shard Allocation
Optimizing insert and query times
can be tricky and require a lot of
attention.
Index Management
Because deletion is an expensive
operation Index management is
required for log analytics solutions
Zone awareness
This is specific for AWS and required to
achieve high availability
Cluster Topology
Elasticsearch clusters require 3
Master nodes, Data nodes and Client
nodes.
Bulk inserts
OptimizationOptimizing insert time and latency
Capacity
provisioningNeed to account for log bursts and be
able to provision enough capacity.
Elasticsearch (2)
Archive (DR)
Snapshot the data to a different
repository for disaster recovery
Mapping
managementMapping conflicts and sync issues
need to be detected and addressed
Monitoring
Marvell does a good job but require
DevOps constant attention
Curator
Remove or optimize old indices
Alias management
For better cluster control you need to
define and use aliases
Data parsing
Extracting values from text messages
and enhancing them with geo user
agent etc.
Logstash
High Availability
Running logstash in a cluster is not
trivial.
Scalability
Dealing with increase of load on the
logstash servers
Burst Protection
Logs tend to be bursty – A special buffer
like Redis, Kafka etc. is required to front
logstash
Rejection from
ElasticsearchElaticsearch rejects about 1% of
messages due to mapping issues –
This needs to be addressed
Configuration
managementA special infrastructure need to be in
place to allow config changes with no
data loss
Security
Kibana by default has no protection.
User authentication is required to be
implemented
Kibana
High Availability
Running Kibana in a cluster for
upgrades and high availability.
Role based access
If you want to restrict access to
certain information this capability
needs to be developed
Alerts
Alerts is not part of the open source.
Anomaly Detection
Basic anomaly detection is missing
from the Kibana
Pre Canned
DashboardsBuilding Dashboards and visualization
in Kibana is tricky and require special
knowledge
Turning ELK Production ready
~ 4-6 weeks of work
Upgrades
Challenging to upgrade – need to be
aware of backward compatibility.
Maintenance
Overall cluster
healthMonitor the health of the
environment
AWS Issues
Dealing with AWS stability issues
Mapping conflicts
Deal with arising mapping conflicts
Personnel
redundancyNeed to have multiple people with
deep knowledge of the stack
Capacity increase
Provision additional capacity and
grow the cluster.
Our Architecture
Ha
Proxy
Listener
Listener
Listener
Listener
Kafka
Log
Engine
S3
Elasticsearch Play
server
Curator
Hot/Cold
migration
DLQ
Alert
Engine
Kibana
Monitoring: ELK, Graphite, Nagios etc.
Shard
optimizer
Log
Engine
Logstash
API Gateway
Cluster
Protec-
tion
Demo
AWS Server Comparison
Machine Number TB/Day
M1.xlarge 4 0.6
i2.xlarge 4 1
C3.8xlarge 6 1.5
C4.2xlarge + 1TB EBS 3 1.3
We’re Hiring
• Technical evangelist
• Business Development
• Marketing
jobs@logz.io
Questions?

More Related Content

PPTX
ELK at LinkedIn - Kafka, scaling, lessons learned
PDF
Search Analytics with ELK (Elastic Stack)
PPTX
Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server
PDF
Kibana + timelion: time series with the elastic stack
PPTX
The Elastic ELK Stack
PDF
Log analysis with the elk stack
PDF
Security monitoring log management-describe logstash,kibana,elastic slidshare
PDF
Meetup070416 Presentations
ELK at LinkedIn - Kafka, scaling, lessons learned
Search Analytics with ELK (Elastic Stack)
Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server
Kibana + timelion: time series with the elastic stack
The Elastic ELK Stack
Log analysis with the elk stack
Security monitoring log management-describe logstash,kibana,elastic slidshare
Meetup070416 Presentations

What's hot (19)

PDF
Elasticsearch in Netflix
PDF
What's new in Elasticsearch v5
PDF
Log analytics with ELK stack
PPTX
Toronto High Scalability meetup - Scaling ELK
PPTX
Sarine's Big Data Journey by Rostislav Aaronov
PDF
Elastic{ON} 2017 Recap
PDF
Scaling ELK Stack - DevOpsDays Singapore
PPTX
Microservices, Continuous Delivery, and Elasticsearch at Capital One
PPTX
The Ultimate Logging Architecture - You KNOW you want it!
PDF
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
PPTX
Elk meetup boston - logz.io
PDF
ストリーミングデータのアドホック分析エンジンの比較
PPTX
Centralised logging with ELK stack
PDF
KafkaとAWS Kinesisの比較
PDF
Replicate Elasticsearch Data with Cross-Cluster Replication (CCR)
PPTX
How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021
PDF
Efficient State Management With Spark 2.0 And Scale-Out Databases
PDF
Queryable State for Kafka Streamsを使ってみた
PPTX
Wikipedia Cloud Search Webinar
Elasticsearch in Netflix
What's new in Elasticsearch v5
Log analytics with ELK stack
Toronto High Scalability meetup - Scaling ELK
Sarine's Big Data Journey by Rostislav Aaronov
Elastic{ON} 2017 Recap
Scaling ELK Stack - DevOpsDays Singapore
Microservices, Continuous Delivery, and Elasticsearch at Capital One
The Ultimate Logging Architecture - You KNOW you want it!
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
Elk meetup boston - logz.io
ストリーミングデータのアドホック分析エンジンの比較
Centralised logging with ELK stack
KafkaとAWS Kinesisの比較
Replicate Elasticsearch Data with Cross-Cluster Replication (CCR)
How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021
Efficient State Management With Spark 2.0 And Scale-Out Databases
Queryable State for Kafka Streamsを使ってみた
Wikipedia Cloud Search Webinar
Ad

Viewers also liked (20)

PDF
Logging with Elasticsearch, Logstash & Kibana
PDF
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
PDF
Introduction To Kibana
PDF
ELK introduction
PDF
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
PPTX
Monitoring Docker with ELK
PDF
Open Source Logging and Monitoring Tools
PPTX
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
PPTX
Scaling an ELK stack at bol.com
PPTX
Elk stack
PDF
An Introduction to event sourcing and CQRS
PPTX
PDF
Duke fuqua marketing forum isbell sep 2014 final
PPTX
Chicago AWS meetup
PDF
Chicago AWS user group meetup - May 2014 at Cohesive
PDF
AWS Chicago user group meetup on June 24, 2014
PDF
Sich selbst verstehen – der ELK-Stack in der Praxis
PPTX
Scott Paddock's AWS Chicago Healthcare slides - 2016
PDF
ELK: a log management framework
Logging with Elasticsearch, Logstash & Kibana
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
Attack monitoring using ElasticSearch Logstash and Kibana
Introduction To Kibana
ELK introduction
Central LogFile Storage. ELK stack Elasticsearch, Logstash and Kibana.
Monitoring Docker with ELK
Open Source Logging and Monitoring Tools
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Scaling an ELK stack at bol.com
Elk stack
An Introduction to event sourcing and CQRS
Duke fuqua marketing forum isbell sep 2014 final
Chicago AWS meetup
Chicago AWS user group meetup - May 2014 at Cohesive
AWS Chicago user group meetup on June 24, 2014
Sich selbst verstehen – der ELK-Stack in der Praxis
Scott Paddock's AWS Chicago Healthcare slides - 2016
ELK: a log management framework
Ad

Similar to Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana) (20)

PPTX
Elk ruminating on logs
PPTX
ELK Ruminating on Logs (Zendcon 2016)
PPTX
Logging using ELK Stack for Microservices
PDF
Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With E...
PDF
Technology behind-real-time-log-analytics
PDF
Lenovo: Elastic Stack Practices in Enterprise Integration
PDF
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
PDF
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
PDF
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
PDF
Music city data Hail Hydrate! from stream to lake
PPTX
Centralized Logging System Using ELK Stack
PPTX
CON6492 - Oracle Database Public Cloud Services v1 1
PDF
OSMC 2023 | Current State of Icinga by Bernd Erk
PDF
Serverless SQL
PPSX
Oracle Exalogic X3-02 Elastic Cloud System
PDF
2015 03-16-elk at-bsides
PDF
Flink in Zalando's world of Microservices
PDF
Flink in Zalando's World of Microservices
PPTX
ELK Stack Online Training - ELK Stack Training.pptx
PPT
Oracle ExaLogic Overview
Elk ruminating on logs
ELK Ruminating on Logs (Zendcon 2016)
Logging using ELK Stack for Microservices
Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With E...
Technology behind-real-time-log-analytics
Lenovo: Elastic Stack Practices in Enterprise Integration
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
Music city data Hail Hydrate! from stream to lake
Centralized Logging System Using ELK Stack
CON6492 - Oracle Database Public Cloud Services v1 1
OSMC 2023 | Current State of Icinga by Bernd Erk
Serverless SQL
Oracle Exalogic X3-02 Elastic Cloud System
2015 03-16-elk at-bsides
Flink in Zalando's world of Microservices
Flink in Zalando's World of Microservices
ELK Stack Online Training - ELK Stack Training.pptx
Oracle ExaLogic Overview

More from Cohesive Networks (20)

PDF
CircleCity Con 2017 - Dwight Koop's talk Cybersecurity for real life: Using t...
PDF
Chris Purrington's talk from CLOUDSEC 2016 "Defense in depth: practical steps...
PDF
Protecting Vital Data With NIST Framework - Patrick Kerpan's Secure260 presen...
PDF
Let’s rethink cloud application security in 2016 - Patrick Kerpan's Secure360...
PDF
The Chicago School of Cybersecurity: A Pragmatic Look at the NIST Cybersecuri...
PDF
Comparison: VNS3 vs Vyatta
PDF
Comparison: VNS3 and Openswan
PDF
Cohesive Networks Support Docs: VNS3 Administration
PDF
Cohesive Networks Support Docs: VNS3 Configuration Guide
PDF
Cohesive Networks Support Docs: VNS3 Configuration for AWS EC2 Classic
PDF
Cohesive Networks Support Docs: VNS3 Configuration for Amazon VPC
PDF
Cohesive Networks Support Docs: VNS3 Configuration in Azure
PDF
Cohesive Networks Support Docs: VNS3 Configuration for CenturyLink Cloud
PDF
Cohesive Networks Support Docs: VNS3 Configuration for IBM Softlayer
PDF
Cohesive Networks Support Docs: VNS3 Configuration for ElasticHosts
PDF
Cohesive Networks Support Docs: VNS3 Configuration for GCE
PDF
Cohesive Networks Support Docs: Welcome to VNS3 3.5
PDF
Cohesive Networks Support Docs: VNS3 Side by Side IPsec Tunnel Guide
PDF
Cohesive networks Support Docs: VNS3 3.5 Upgrade Guide
PDF
Cohesive Networks Support Docs: VNS3 3.5 Container System Add-Ons
CircleCity Con 2017 - Dwight Koop's talk Cybersecurity for real life: Using t...
Chris Purrington's talk from CLOUDSEC 2016 "Defense in depth: practical steps...
Protecting Vital Data With NIST Framework - Patrick Kerpan's Secure260 presen...
Let’s rethink cloud application security in 2016 - Patrick Kerpan's Secure360...
The Chicago School of Cybersecurity: A Pragmatic Look at the NIST Cybersecuri...
Comparison: VNS3 vs Vyatta
Comparison: VNS3 and Openswan
Cohesive Networks Support Docs: VNS3 Administration
Cohesive Networks Support Docs: VNS3 Configuration Guide
Cohesive Networks Support Docs: VNS3 Configuration for AWS EC2 Classic
Cohesive Networks Support Docs: VNS3 Configuration for Amazon VPC
Cohesive Networks Support Docs: VNS3 Configuration in Azure
Cohesive Networks Support Docs: VNS3 Configuration for CenturyLink Cloud
Cohesive Networks Support Docs: VNS3 Configuration for IBM Softlayer
Cohesive Networks Support Docs: VNS3 Configuration for ElasticHosts
Cohesive Networks Support Docs: VNS3 Configuration for GCE
Cohesive Networks Support Docs: Welcome to VNS3 3.5
Cohesive Networks Support Docs: VNS3 Side by Side IPsec Tunnel Guide
Cohesive networks Support Docs: VNS3 3.5 Upgrade Guide
Cohesive Networks Support Docs: VNS3 3.5 Container System Add-Ons

Recently uploaded (20)

PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Machine learning based COVID-19 study performance prediction
PDF
Electronic commerce courselecture one. Pdf
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Approach and Philosophy of On baking technology
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PPT
Teaching material agriculture food technology
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Empathic Computing: Creating Shared Understanding
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
Building Integrated photovoltaic BIPV_UPV.pdf
Per capita expenditure prediction using model stacking based on satellite ima...
Machine learning based COVID-19 study performance prediction
Electronic commerce courselecture one. Pdf
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Diabetes mellitus diagnosis method based random forest with bat algorithm
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
The Rise and Fall of 3GPP – Time for a Sabbatical?
Approach and Philosophy of On baking technology
Advanced methodologies resolving dimensionality complications for autism neur...
Teaching material agriculture food technology
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Empathic Computing: Creating Shared Understanding
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
MYSQL Presentation for SQL database connectivity
Chapter 3 Spatial Domain Image Processing.pdf
Understanding_Digital_Forensics_Presentation.pptx
NewMind AI Weekly Chronicles - August'25 Week I
20250228 LYD VKU AI Blended-Learning.pptx

Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)

  • 2. Who am I Asaf Yigal Co-Founder and VP Product @logz.io Email: asaf@logz.io Twitter @asafyigal
  • 3. Agenda • Why do we need Log analytics? • Intro to ELK • What is Logz.io • Installing ELK on your own • Our Architecture • EC2 machine comparison
  • 4. Why do we need Log analytics?
  • 5. Werner Vogels AWS CTO “Log Analytics is Fundamental for Building Cloud Applications”
  • 7. Log driven development • Errors, Warnings and exceptions • Metrics • Alerts • Dashboard
  • 10. *based on Logz.io research The Market is Dominated by Open Source Solutions Over the past 3 years, the market shifted attention from proprietary to open source ELK Stack, 400,000+ companies Splunk, Sumo Logic, Loggly, - 20,000 companies Graphite has > 1M companies using it
  • 12. Intro to ELK Logstash •Streaming data digestion •Time normalization •Field extraction Elasticsearch •Schema-less search DB •Highly scalable Kibana •Visualization
  • 13. Open source ELK +/- Simple and beautifulIt’s simple to get started and play with ELK and the UI is just beautiful Open Source The largest user base with a vibrant open source community that supports and improves the product Fast. Very fast. Built on the Elasticsearch search engine, ELK provide blazing quick responses even when searching through millions of documents Hard to Scale Data piles up and organization experience usage bursts. It’s super-complex building elastic ELK deployments that can scale up and down Poor Security Logs include sensitive data and open source ELK offers no real security solution, from authentication to role based access Not Production Ready Building production ready ELK deployment is a great challenge organization face. With hundreds of different configurations and support matrix, making sure it’s always up is difficult
  • 14. Up and running in minutesSign up in and get insights into your data in minutes Logz.io Enterprise ELK Cloud Service Production ready Predefined and community designed dashboard, visualization and alerts are all bundled and ready to provide insights Infinitely scalable Ship as much data as you want whenever you want Alerts Unique Alerts system proprietary built on top of open source ELK transform the ELK into a proactive system Highly Available Data and entire data ingestion pipeline can sustain downtime in full datacenter without losing data or service Advanced Security 360 degrees security with role based access and multi-layer security
  • 15. Installing ELK on your own
  • 16. Prototype • Installing ELK stack on a single server – 1hr • Shipping one type of log – 1hr • Log parsing – 2 hr • Building Kibana Dashboard – 2hr • 6 hours to get a simple Prototype
  • 18. OS Level OptimizationElasticsearch require a lot of OS level optimization in order to run properly. Elasticsearch Shard Allocation Optimizing insert and query times can be tricky and require a lot of attention. Index Management Because deletion is an expensive operation Index management is required for log analytics solutions Zone awareness This is specific for AWS and required to achieve high availability Cluster Topology Elasticsearch clusters require 3 Master nodes, Data nodes and Client nodes. Bulk inserts OptimizationOptimizing insert time and latency
  • 19. Capacity provisioningNeed to account for log bursts and be able to provision enough capacity. Elasticsearch (2) Archive (DR) Snapshot the data to a different repository for disaster recovery Mapping managementMapping conflicts and sync issues need to be detected and addressed Monitoring Marvell does a good job but require DevOps constant attention Curator Remove or optimize old indices Alias management For better cluster control you need to define and use aliases
  • 20. Data parsing Extracting values from text messages and enhancing them with geo user agent etc. Logstash High Availability Running logstash in a cluster is not trivial. Scalability Dealing with increase of load on the logstash servers Burst Protection Logs tend to be bursty – A special buffer like Redis, Kafka etc. is required to front logstash Rejection from ElasticsearchElaticsearch rejects about 1% of messages due to mapping issues – This needs to be addressed Configuration managementA special infrastructure need to be in place to allow config changes with no data loss
  • 21. Security Kibana by default has no protection. User authentication is required to be implemented Kibana High Availability Running Kibana in a cluster for upgrades and high availability. Role based access If you want to restrict access to certain information this capability needs to be developed Alerts Alerts is not part of the open source. Anomaly Detection Basic anomaly detection is missing from the Kibana Pre Canned DashboardsBuilding Dashboards and visualization in Kibana is tricky and require special knowledge
  • 22. Turning ELK Production ready ~ 4-6 weeks of work
  • 23. Upgrades Challenging to upgrade – need to be aware of backward compatibility. Maintenance Overall cluster healthMonitor the health of the environment AWS Issues Dealing with AWS stability issues Mapping conflicts Deal with arising mapping conflicts Personnel redundancyNeed to have multiple people with deep knowledge of the stack Capacity increase Provision additional capacity and grow the cluster.
  • 26. Demo
  • 27. AWS Server Comparison Machine Number TB/Day M1.xlarge 4 0.6 i2.xlarge 4 1 C3.8xlarge 6 1.5 C4.2xlarge + 1TB EBS 3 1.3
  • 28. We’re Hiring • Technical evangelist • Business Development • Marketing jobs@logz.io