SlideShare a Scribd company logo

From Zero to Hero - Centralized Logging with Logstash & Elasticsearch

Sematext Group, Inc. , profile picture
Sematext Group, Inc.

The document discusses log management and analysis using Elasticsearch and Logstash, presenting various configurations and best practices for effective log centralization and searching. It emphasizes the benefits of real-time data insight, and includes installation instructions, configuration settings, and sample log processing examples. Additionally, it promotes related services and job opportunities at Sematext.

Technology
1 of 73
Downloaded 432 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
From Zero to Hero Rafał Kuć – Sematext Group, Inc. @kucrafal @sematext sematext.com Easy log centralization with Logstash & Elasticsearch
About me… Sematext consultant & engineer Solr.pl co-founder Father and husband 
The problem
The problem Log Log Log Log Log Log Log Log Log
Let’s find something http://www.likesbooks.com/aarafterhours/?p=750
The solution Log Log Log Log Log Log Log Log
Available tools
Available tools …
But why search? Easy to find related data
But why search? Easy to find related data Fast and accurate
But why search? Easy to find related data Fast and accurate Real time data insight and analysis
Why Elasticsearch? Reasonable defaults Distributed by design http://www.dailypets.co.uk/2007/06/17/kittens-rest-at-half-time/
Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz
Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz $ tar –xvf elasticsearch-1.3.2.tar.gz $ elasticsearch-1.3.2/bin/elasticsearch
Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz $ tar –xvf elasticsearch-1.3.2.tar.gz $ elasticsearch-1.3.2/bin/elasticsearch
Scalable
Scalable
Scalable
Configuration - stability
Configuration - stability
Configuration - stability
Configuration - stability minimum_master_nodes = N/2 + 1
Configuration - stability Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1
Thread pools
Thread pools Use fixed Set size Set queue
Thread pools threadpool.search.type threadpool.search.size threadpool.search.queue_size threadpool.index.type threadpool.index.size threadpool.index.queue_size threadpool.bulk.type threadpool.bulk.size threadpool.bulk.queue_size Use fixed Set size Set queue
Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead 40% Xmx 1
Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead 60% Xmx 1.03 40% Xmx 1
Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit 70% Xmx 60% Xmx 1.03 40% Xmx 1
Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit indices.fielddata.cache.size unbounded 70% Xmx 60% Xmx 1.03 40% Xmx 1
Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit indices.fielddata.cache.size indices.cache.filter.size unbounded 10% 70% Xmx 60% Xmx 1.03 40% Xmx 1
Configuration - indexing Log
Configuration - indexing Log
Configuration - indexing Log Log Log Log Log Log Log Log Log Use Bulk! Or UDP Bulk!
Configuration - indexing Log Log Log Log Log Log Log Log Log index.translog.flush_threshold_ops index.translog.flush_threshold_size unlimited 200mb Use Bulk! Or UDP Bulk!
Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ 5s refresh -> 2.5K logs/sec
Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ 5s refresh -> 2.5K logs/sec 30s refresh -> 3.4K logs/sec
Data volume under control 2014-09-24
Data volume under control 2014-09-24 TODAY
Data volume under control 2014-09-24 TODAY WEEK
Data volume under control 2014-09-24 2014-09-25 TODAY WEEK
Data volume under control 2014-09-24 2014-09-25 2014-09-26 TODAY WEEK
Monitoring
Monitoring
Monitoring
SPM http://sematext.com/spm/
SPM http://sematext.com/spm/
SPM http://sematext.com/spm/
SPM http://sematext.com/spm/
Here comes Logstash Unstructured
Here comes Logstash Unstructured
Here comes Logstash Unstructured Documents
Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" { "host" : "127.0.0.1", "@timestamp" : "2014-02-05T17:11:55+0000", ... "verb" : "GET" }
How does it look?
Of course you can scale
Logstash input input { file { path => "/var/log/apache/apache.log" type => "access_apache_log" start_position => "beginning" } }
Grok filter { if [type] == "access_apache_log" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } }
Logstash output output { elasticsearch { host => "localhost" port => 9200 index => "logs_%{+YYYY.MM.dd}" protocol => "http" manage_template => true } }
Sample Logstash-forwarder config { "network": { "servers": [ "localhost:5043" ], "timeout": 15 }, "files": [ { "paths": [ "/var/log/apache/apache*.log" ], "fields": { "type": "access_apache_log" } } ] }
Sample Logstash-forwarder config { "network": { "servers": [ "localhost:5043" ], "timeout": 15 }, "files": [ { "paths": [ "/var/log/apache/apache*.log" ], "fields": { "type": "access_apache_log" } } ] } Logstash side: input { lumberjack { port => 5043 type => "access_apache_log" } }
Let’s try it $ bin/logstash –f logstash-filter.conf
Let’s try it $ bin/logstash –f logstash-filter.conf $ curl 'localhost:9200/logs_2014-09-26/_search?pretty'
Let’s try it $ bin/logstash –f logstash-filter.conf $ curl 'localhost:9200/logs_2014-09-26/_search?pretty' { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "logs", "_type" : "access_apache_log", "_id" : "SI0BZw8BQ0uQNPtk9zfoOQ", "_score" : 1.0, "_source":{"message":"71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"","@version":"1","@timestamp":"2014-09-11T10:21:04.403Z","type":"access_apache_log","host":"developer-vb","path":"/home/gro/devops/apache3.log","clientip":"71.141.244.242","ident":"- ","auth":"kurt","timestamp":"18/May/2011:01:48:10 -0700","verb":"GET","request":"/admin","httpversion":"1.1","response":"301","bytes":"566","referrer":""-"","agent":""Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3""} }, { "_index" : "logs", "_type" : "access_apache_log", "_id" : "zyOc53uwQkegOQr-a3hwIQ", "_score" : 1.0, "_source":{"message":"98.83.179.51 - - [18/May/2011:19:35:08 -0700] "GET /css/main.css HTTP/1.1" 200 1837 "http://www.safesand.com/information.htm" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"","@version":"1","@timestamp":"2014-09-11T10:21:04.405Z","type":"access_apache_log","host":"developer- vb","path":"/home/gro/devops/apache3.log","clientip":"98.83.179.51","ident":"-","auth":"-","timestamp":"18/May/2011:19:35:08 - 0700","verb":"GET","request":"/css/main.css","httpversion":"1.1","response":"200","bytes":"1837","referrer":""http://www.safesand.com/information.htm"","agent":""Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1""} }, { "_index" : "logs", "_type" : "access_apache_log", "_id" : "evP0I--3TWOlDsQzalQtAw", "_score" : 1.0, "_source":{"message":"134.39.72.245 - - [18/May/2011:12:40:18 -0700] "GET /favicon.ico HTTP/1.1" 200 1189 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)"","@version":"1","@timestamp":"2014-09-11T10:21:04.404Z","type":"access_apache_log","host":"developer- vb","path":"/home/gro/devops/apache3.log","clientip":"134.39.72.245","ident":"-","auth":"-","timestamp":"18/May/2011:12:40:18 - 0700","verb":"GET","request":"/favicon.ico","httpversion":"1.1","response":"200","bytes":"1189","referrer":""-"","agent":""Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)""} } ] } }
Here comes Kibana
Looking for SaaS – Go Logsene http://sematext.com/logsene
Looking for SaaS – Go Logsene http://sematext.com/logsene
Logstash + Logsene in action output { elasticsearch { host => "logsene-receiver.sematext.com" port => 80 index => "YOUR_TOKEN" protocol => "http" manage_template => false } } http://sematext.com/logsene
Short summary http://www.soothetube.com/2013/12/29/thats-all-folks/
We Are Hiring ! Dig Search ? Dig Analytics ? Dig Big Data ? Dig Performance ? Dig Logging ? Dig working with and in open – source ? We’re hiring world – wide ! http://sematext.com/about/jobs.html
Rafał Kuć @kucrafal rafal.kuc@sematext.com Sematext @sematext http://sematext.com http://blog.sematext.com Thank You !

More Related Content

PDF
On Centralizing Logs
Sematext Group, Inc.
 
PPTX
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Data Con LA
 
PDF
Application Logging With Logstash
benwaine
 
PPTX
MySQL Slow Query log Monitoring using Beats & ELK
YoungHeon (Roy) Kim
 
PDF
Tuning Solr for Logs
Sematext Group, Inc.
 
PDF
Solr for Indexing and Searching Logs
Sematext Group, Inc.
 
PPT
ELK stack at weibo.com
琛琳 饶
 
PPT
Logstash
琛琳 饶
 
On Centralizing Logs
Sematext Group, Inc.
 
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Data Con LA
 
Application Logging With Logstash
benwaine
 
MySQL Slow Query log Monitoring using Beats & ELK
YoungHeon (Roy) Kim
 
Tuning Solr for Logs
Sematext Group, Inc.
 
Solr for Indexing and Searching Logs
Sematext Group, Inc.
 
ELK stack at weibo.com
琛琳 饶
 
Logstash
琛琳 饶
 

What's hot (20)

PDF
Logstash family introduction
Owen Wu
 
PPTX
Managing Your Security Logs with Elasticsearch
Vic Hargrave
 
PDF
Tuning Solr & Pipeline for Logs
Sematext Group, Inc.
 
PDF
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
PDF
Dive into Fluentd plugin v0.12
N Masahiro
 
PPT
{{more}} Kibana4
琛琳 饶
 
PDF
Fluentd v1.0 in a nutshell
N Masahiro
 
PDF
Fluentd v0.12 master guide
N Masahiro
 
PPTX
Monitoring Docker with ELK
Daniel Berman
 
PPT
Large Scale Log collection using LogStash & mongoDB
Gaurav Bhardwaj
 
PDF
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Zabbix
 
PDF
elk_stack_alexander_szalonnas
Alexander Szalonnas
 
PPTX
The tale of 100 cve's
Prajal Kulkarni
 
PDF
NginX - good practices, tips and advanced techniques
Claudio Borges
 
PPT
Learn ELK in docker
Larry Cai
 
PDF
Securing Prometheus exporters using HashiCorp Vault
Bram Vogelaar
 
PPT
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Logging logs with Logstash - Devops MK 10-02-2016
Steve Howe
 
PDF
Aaron Mildenstein - Using Logstash with Zabbix
Zabbix
 
PDF
Fluentd v1 and Roadmap
Treasure Data, Inc.
 
Logstash family introduction
Owen Wu
 
Managing Your Security Logs with Elasticsearch
Vic Hargrave
 
Tuning Solr & Pipeline for Logs
Sematext Group, Inc.
 
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
Dive into Fluentd plugin v0.12
N Masahiro
 
{{more}} Kibana4
琛琳 饶
 
Fluentd v1.0 in a nutshell
N Masahiro
 
Fluentd v0.12 master guide
N Masahiro
 
Monitoring Docker with ELK
Daniel Berman
 
Large Scale Log collection using LogStash & mongoDB
Gaurav Bhardwaj
 
Rihards Olups - Encrypting Daemon Traffic With Zabbix 3.0
Zabbix
 
elk_stack_alexander_szalonnas
Alexander Szalonnas
 
The tale of 100 cve's
Prajal Kulkarni
 
NginX - good practices, tips and advanced techniques
Claudio Borges
 
Learn ELK in docker
Larry Cai
 
Securing Prometheus exporters using HashiCorp Vault
Bram Vogelaar
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Logging logs with Logstash - Devops MK 10-02-2016
Steve Howe
 
Aaron Mildenstein - Using Logstash with Zabbix
Zabbix
 
Fluentd v1 and Roadmap
Treasure Data, Inc.
 
Ad

Viewers also liked (6)

PDF
Application Logging With The ELK Stack
benwaine
 
PDF
Fluentd and docker monitoring
Vinay Krishna
 
PDF
Fluentd and PHP
chobi e
 
PDF
Integrando Redis en aplicaciones Symfony2
Ronny López
 
PDF
Fluentd vs. Logstash for OpenStack Log Management
NTT Communications Technology Development
 
PDF
Logging with Elasticsearch, Logstash & Kibana
Amazee Labs
 
Application Logging With The ELK Stack
benwaine
 
Fluentd and docker monitoring
Vinay Krishna
 
Fluentd and PHP
chobi e
 
Integrando Redis en aplicaciones Symfony2
Ronny López
 
Fluentd vs. Logstash for OpenStack Log Management
NTT Communications Technology Development
 
Logging with Elasticsearch, Logstash & Kibana
Amazee Labs
 
Ad

Similar to From Zero to Hero - Centralized Logging with Logstash & Elasticsearch (20)

KEY
London devops logging
Tomas Doran
 
PPTX
Tuning Elasticsearch Indexing Pipeline for Logs
Sematext Group, Inc.
 
PDF
ELK stack introduction
abenyeung1
 
PPTX
Centralized log-management-with-elastic-stack
Rich Lee
 
PPTX
Scaling an ELK stack at bol.com
Renzo Tomà
 
PDF
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
PPT
How ElasticSearch lives in my DevOps life
琛琳 饶
 
PDF
Tuning Solr for Logs: Presented by Radu Gheorghe, Sematext
Lucidworks
 
PDF
Managing Your Content with Elasticsearch
Samantha Quiñones
 
KEY
Zero mq logs
Tomas Doran
 
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
PPTX
Search and analyze data in real time
Rohit Kalsarpe
 
PDF
VictoriaLogs: Open Source Log Management System - Preview
VictoriaMetrics
 
PDF
Brisbane DevOps Meetup - Logstash
pczarkowski
 
PDF
Es part 2 pdf no build
Erik Rose
 
PPT
'Scalable Logging and Analytics with LogStash'
Cloud Elements
 
PDF
ELK: Moose-ively scaling your log system
Avleen Vig
 
KEY
Message:Passing - lpw 2012
Tomas Doran
 
PDF
Log analysis with the elk stack
Vikrant Chauhan
 
PPTX
Toronto High Scalability meetup - Scaling ELK
Andrew Trossman
 
London devops logging
Tomas Doran
 
Tuning Elasticsearch Indexing Pipeline for Logs
Sematext Group, Inc.
 
ELK stack introduction
abenyeung1
 
Centralized log-management-with-elastic-stack
Rich Lee
 
Scaling an ELK stack at bol.com
Renzo Tomà
 
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
How ElasticSearch lives in my DevOps life
琛琳 饶
 
Tuning Solr for Logs: Presented by Radu Gheorghe, Sematext
Lucidworks
 
Managing Your Content with Elasticsearch
Samantha Quiñones
 
Zero mq logs
Tomas Doran
 
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
Search and analyze data in real time
Rohit Kalsarpe
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaMetrics
 
Brisbane DevOps Meetup - Logstash
pczarkowski
 
Es part 2 pdf no build
Erik Rose
 
'Scalable Logging and Analytics with LogStash'
Cloud Elements
 
ELK: Moose-ively scaling your log system
Avleen Vig
 
Message:Passing - lpw 2012
Tomas Doran
 
Log analysis with the elk stack
Vikrant Chauhan
 
Toronto High Scalability meetup - Scaling ELK
Andrew Trossman
 

More from Sematext Group, Inc. (20)

PDF
Tweaking the Base Score: Lucene/Solr Similarities Explained
Sematext Group, Inc.
 
PDF
OOPs, OOMs, oh my! Containerizing JVM apps
Sematext Group, Inc.
 
PPTX
Is observability good for your brain?
Sematext Group, Inc.
 
PDF
Introducing log analysis to your organization
Sematext Group, Inc.
 
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
Sematext Group, Inc.
 
PDF
Solr on Docker - the Good, the Bad and the Ugly
Sematext Group, Inc.
 
PDF
Monitoring and Log Management for
Sematext Group, Inc.
 
PDF
Introduction to solr
Sematext Group, Inc.
 
PDF
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
Sematext Group, Inc.
 
PDF
How to Run Solr on Docker and Why
Sematext Group, Inc.
 
PPTX
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Top Node.js Metrics to Watch
Sematext Group, Inc.
 
PDF
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
Sematext Group, Inc.
 
PDF
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Sematext Group, Inc.
 
PDF
Docker Logging Webinar
Sematext Group, Inc.
 
PDF
Docker Monitoring Webinar
Sematext Group, Inc.
 
PDF
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Sematext Group, Inc.
 
PDF
Side by Side with Elasticsearch & Solr, Part 2
Sematext Group, Inc.
 
PDF
Solr Anti Patterns
Sematext Group, Inc.
 
PDF
(Elastic)search in big data
Sematext Group, Inc.
 
Tweaking the Base Score: Lucene/Solr Similarities Explained
Sematext Group, Inc.
 
OOPs, OOMs, oh my! Containerizing JVM apps
Sematext Group, Inc.
 
Is observability good for your brain?
Sematext Group, Inc.
 
Introducing log analysis to your organization
Sematext Group, Inc.
 
Solr Search Engine: Optimize Is (Not) Bad for You
Sematext Group, Inc.
 
Solr on Docker - the Good, the Bad and the Ugly
Sematext Group, Inc.
 
Monitoring and Log Management for
Sematext Group, Inc.
 
Introduction to solr
Sematext Group, Inc.
 
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
Sematext Group, Inc.
 
How to Run Solr on Docker and Why
Sematext Group, Inc.
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Top Node.js Metrics to Watch
Sematext Group, Inc.
 
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
Sematext Group, Inc.
 
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Sematext Group, Inc.
 
Docker Logging Webinar
Sematext Group, Inc.
 
Docker Monitoring Webinar
Sematext Group, Inc.
 
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Sematext Group, Inc.
 
Side by Side with Elasticsearch & Solr, Part 2
Sematext Group, Inc.
 
Solr Anti Patterns
Sematext Group, Inc.
 
(Elastic)search in big data
Sematext Group, Inc.
 

Recently uploaded (20)

PDF
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
Advanced Soft Computing BINUS July 2025.pdf
University of Hertfordshire
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Cloud computing and distributed systems.
kkkkkhan
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Advanced Soft Computing BINUS July 2025.pdf
University of Hertfordshire
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Advanced IT Governance
University of Hertfordshire
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 

From Zero to Hero - Centralized Logging with Logstash & Elasticsearch

  • 1. From Zero to Hero Rafał Kuć – Sematext Group, Inc. @kucrafal @sematext sematext.com Easy log centralization with Logstash & Elasticsearch
  • 2. About me… Sematext consultant & engineer Solr.pl co-founder Father and husband 
  • 4. The problem Log Log Log Log Log Log Log Log Log
  • 5. Let’s find something http://www.likesbooks.com/aarafterhours/?p=750
  • 6. The solution Log Log Log Log Log Log Log Log
  • 9. But why search? Easy to find related data
  • 10. But why search? Easy to find related data Fast and accurate
  • 11. But why search? Easy to find related data Fast and accurate Real time data insight and analysis
  • 12. Why Elasticsearch? Reasonable defaults Distributed by design http://www.dailypets.co.uk/2007/06/17/kittens-rest-at-half-time/
  • 13. Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz
  • 14. Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz $ tar –xvf elasticsearch-1.3.2.tar.gz $ elasticsearch-1.3.2/bin/elasticsearch
  • 15. Installation $ wget --no-check-certificate https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz $ tar –xvf elasticsearch-1.3.2.tar.gz $ elasticsearch-1.3.2/bin/elasticsearch
  • 22. Configuration - stability minimum_master_nodes = N/2 + 1
  • 23. Configuration - stability Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1
  • 25. Thread pools Use fixed Set size Set queue
  • 26. Thread pools threadpool.search.type threadpool.search.size threadpool.search.queue_size threadpool.index.type threadpool.index.size threadpool.index.queue_size threadpool.bulk.type threadpool.bulk.size threadpool.bulk.queue_size Use fixed Set size Set queue
  • 27. Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead 40% Xmx 1
  • 28. Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead 60% Xmx 1.03 40% Xmx 1
  • 29. Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit 70% Xmx 60% Xmx 1.03 40% Xmx 1
  • 30. Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit indices.fielddata.cache.size unbounded 70% Xmx 60% Xmx 1.03 40% Xmx 1
  • 31. Circuit breakers, caches == no OOM indices.breaker.fielddata.limit indices.breaker.fielddata.overhead indices.breaker.request.limit indices.breaker.request.overhead indices.breaker.total.limit indices.fielddata.cache.size indices.cache.filter.size unbounded 10% 70% Xmx 60% Xmx 1.03 40% Xmx 1
  • 34. Configuration - indexing Log Log Log Log Log Log Log Log Log Use Bulk! Or UDP Bulk!
  • 35. Configuration - indexing Log Log Log Log Log Log Log Log Log index.translog.flush_threshold_ops index.translog.flush_threshold_size unlimited 200mb Use Bulk! Or UDP Bulk!
  • 36. Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
  • 37. Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ 5s refresh -> 2.5K logs/sec
  • 38. Refresh when needed 1s refresh -> 2K logs/sec http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ 5s refresh -> 2.5K logs/sec 30s refresh -> 3.4K logs/sec
  • 39. Data volume under control 2014-09-24
  • 40. Data volume under control 2014-09-24 TODAY
  • 41. Data volume under control 2014-09-24 TODAY WEEK
  • 42. Data volume under control 2014-09-24 2014-09-25 TODAY WEEK
  • 43. Data volume under control 2014-09-24 2014-09-25 2014-09-26 TODAY WEEK
  • 51. Here comes Logstash Unstructured
  • 52. Here comes Logstash Unstructured
  • 53. Here comes Logstash Unstructured Documents
  • 54. Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
  • 55. Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
  • 56. Processing example 127.0.0.1 - - [05/Feb/2014:17:11:55 +0000] "GET /css/main.css HTTP/1.1" 200 140 "http://www.onet.pl" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" { "host" : "127.0.0.1", "@timestamp" : "2014-02-05T17:11:55+0000", ... "verb" : "GET" }
  • 57. How does it look?
  • 58. Of course you can scale
  • 59. Logstash input input { file { path => "/var/log/apache/apache.log" type => "access_apache_log" start_position => "beginning" } }
  • 60. Grok filter { if [type] == "access_apache_log" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } }
  • 61. Logstash output output { elasticsearch { host => "localhost" port => 9200 index => "logs_%{+YYYY.MM.dd}" protocol => "http" manage_template => true } }
  • 62. Sample Logstash-forwarder config { "network": { "servers": [ "localhost:5043" ], "timeout": 15 }, "files": [ { "paths": [ "/var/log/apache/apache*.log" ], "fields": { "type": "access_apache_log" } } ] }
  • 63. Sample Logstash-forwarder config { "network": { "servers": [ "localhost:5043" ], "timeout": 15 }, "files": [ { "paths": [ "/var/log/apache/apache*.log" ], "fields": { "type": "access_apache_log" } } ] } Logstash side: input { lumberjack { port => 5043 type => "access_apache_log" } }
  • 64. Let’s try it $ bin/logstash –f logstash-filter.conf
  • 65. Let’s try it $ bin/logstash –f logstash-filter.conf $ curl 'localhost:9200/logs_2014-09-26/_search?pretty'
  • 66. Let’s try it $ bin/logstash –f logstash-filter.conf $ curl 'localhost:9200/logs_2014-09-26/_search?pretty' { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "logs", "_type" : "access_apache_log", "_id" : "SI0BZw8BQ0uQNPtk9zfoOQ", "_score" : 1.0, "_source":{"message":"71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"","@version":"1","@timestamp":"2014-09-11T10:21:04.403Z","type":"access_apache_log","host":"developer-vb","path":"/home/gro/devops/apache3.log","clientip":"71.141.244.242","ident":"- ","auth":"kurt","timestamp":"18/May/2011:01:48:10 -0700","verb":"GET","request":"/admin","httpversion":"1.1","response":"301","bytes":"566","referrer":""-"","agent":""Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3""} }, { "_index" : "logs", "_type" : "access_apache_log", "_id" : "zyOc53uwQkegOQr-a3hwIQ", "_score" : 1.0, "_source":{"message":"98.83.179.51 - - [18/May/2011:19:35:08 -0700] "GET /css/main.css HTTP/1.1" 200 1837 "http://www.safesand.com/information.htm" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"","@version":"1","@timestamp":"2014-09-11T10:21:04.405Z","type":"access_apache_log","host":"developer- vb","path":"/home/gro/devops/apache3.log","clientip":"98.83.179.51","ident":"-","auth":"-","timestamp":"18/May/2011:19:35:08 - 0700","verb":"GET","request":"/css/main.css","httpversion":"1.1","response":"200","bytes":"1837","referrer":""http://www.safesand.com/information.htm"","agent":""Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1""} }, { "_index" : "logs", "_type" : "access_apache_log", "_id" : "evP0I--3TWOlDsQzalQtAw", "_score" : 1.0, "_source":{"message":"134.39.72.245 - - [18/May/2011:12:40:18 -0700] "GET /favicon.ico HTTP/1.1" 200 1189 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)"","@version":"1","@timestamp":"2014-09-11T10:21:04.404Z","type":"access_apache_log","host":"developer- vb","path":"/home/gro/devops/apache3.log","clientip":"134.39.72.245","ident":"-","auth":"-","timestamp":"18/May/2011:12:40:18 - 0700","verb":"GET","request":"/favicon.ico","httpversion":"1.1","response":"200","bytes":"1189","referrer":""-"","agent":""Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)""} } ] } }
  • 68. Looking for SaaS – Go Logsene http://sematext.com/logsene
  • 69. Looking for SaaS – Go Logsene http://sematext.com/logsene
  • 70. Logstash + Logsene in action output { elasticsearch { host => "logsene-receiver.sematext.com" port => 80 index => "YOUR_TOKEN" protocol => "http" manage_template => false } } http://sematext.com/logsene
  • 72. We Are Hiring ! Dig Search ? Dig Analytics ? Dig Big Data ? Dig Performance ? Dig Logging ? Dig working with and in open – source ? We’re hiring world – wide ! http://sematext.com/about/jobs.html
  • 73. Rafał Kuć @kucrafal rafal.kuc@sematext.com Sematext @sematext http://sematext.com http://blog.sematext.com Thank You !