Kibana+ElasticSearch+LogStash to handle Log messages on Prod servers
Download as PPTX, PDF0 likes391 views
The document provides an overview of logging and the Elastic Stack (Elasticsearch, Logstash, Kibana) for log analysis. It discusses what logs are, popular logging processors like Loggly and Splunk, and the key components of the Elastic Stack - Elasticsearch for storage, Logstash for processing, and Kibana for visualization. It also gives a live demo example of using Logstash to ingest application logs and explore them in Kibana.
Kibana+ElasticSearch+LogStash to handle Log messages on Prod servers
- 1. Kibana + ElasticSearch + LogStash By Dmitriy Mustafin JavaMeetup handle Log messages on Prod servers
- 2. My experience - I started with machine code and punched cards. I used to write on Assembler, then on C and Pascal, С++, Delphi, C#, Javascript, and some other scripting languages. Now I mostly write on Java, and I’m pretty much happy with that. - DOS (it was a wonderful to have 21h interrupt), OS/2, QNX, Windows (my favorite so far), Linux (and Yocto as well), MacOS/iOS. - I studied at university but I still haven't defended my thesis (and I'm not planning on doing this anytime soon). I keep learning new things during my work. - I was writing, designing, engineering, managing and solving problems. - Married, children, cat.
- 3. HYS Enterprise is a Dutch software development company with more than 200 talented engineers from all over the world hys-enterprise.com
- 4. Agenda ● What is log and what’s this for? ● How can we do log totally senseless? ● Logging levels or how not to drown in gigs of data ● Most popular logging processors ● Elastic Stack as a result of smoking in the Netherlands ● This beautiful Kibana ● This nimble and gluttonous ElasticSearch ● This terrifying LogStash ● LogStash: real-life example (or even live demo) ● How can we avoid pitfalls of LogStash? ● ElasticBeats or “What else do you want from me?”
- 5. What is log and what’s this for? In computing, a log file is a file that records either events that occur in an operating system or other software runs, or messages between different users of a communication software. Logging is the act of keeping a log. In the simplest case, messages are written to a single log file. Wikipedia
- 6. How can we do log totally senseless? Harmful advices: ● Do not use log-files! ● Never put datetime, log level, process id, thread id, specific entity grouping, message itself into log message ● Never make file rolling! ● 15 GB log-file is ok ● 150+ GB log-file is perfect! ● Never zip old files ● Keep log files no more than 1 day (max 2 days) ● Nobody should to know about log-files! ● Never analyze log-files! ● Never put live data on big screen in developers room!
- 7. Logging levels or how not to drown in gigs of data + custom log levels Standard log levels built-in to Log4J 2 ™ intLevel OFF 0 FATAL 100 ERROR 200 WARN 300 INFO 400 DEBUG 500 TRACE 600 ALL Int max
- 8. Most popular logging processors Top 10 (?) Log Analysis Tools by KeyCDN: 1. Loggly 2. Logentries 3. GoAccess 4. logz.io 5. Graylog 6. Splunk 7. Logmatic.io 8. Logstash 9. Sumo Logic 10. Papertrail 11. Fluentd
- 9. Elastic Stack as a result of smoking in the Netherlands ● Original author: Shay Banon ● Stable release: 6.4.2 / October 2, 2018 * ● Repository: github.com/elastic/elasticsearch ● Written in Java ● Operating system: Cross-platform ● Type: Search and index ● License: Apache License 2.0 ● Website: www.elastic.co/products/elasticsearch
- 10. Elastic Stack as a result of smoking in the Netherlands - cont. ● Shay Banon (Compass) 2004 - Downloads: 0 ● June 2012 - Downloads: <16,000 ● July 2015 - Downloads: 36,431,145 ● October 2015 - Downloads: 44,378,846 ● October 2016 - Downloads: 91,183,928 ● May 2017 - Downloads: 137,715,884 ● October 2017 - Downloads: 192,865,831 ● February 2018 - Downloads: 267,972,265 18 offices in Europe, America, Asia
- 11. This beautiful Kibana Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers. https://www.elastic.co/products/kibana ● Web UI ● Useful search and filtering ● Visualisations ● Dashboards ● Compute fields
- 14. This nimble and gluttonous ElasticSearch ElasticSearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. https://www.elastic.co/products/elasticsearc h ● ElasticSearch Is Fast. Really, Really Fast. ● Run It on Your Laptop or Hundreds of Servers with Petabytes of Data. ● Interact with ElasticSearch in the Programming Language You Choose. ● Extend ElasticSearch.
- 15. This terrifying LogStash Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash”. https://www.elastic.co/products/logstas h ● Takes 1GB or RAM (Java!) ● Parsing and computing @ onboard machine ● Network consumption is tens of MBps ● Looooong start (Java!) ● GrOk and RegExp ● Not easy to debug the script
- 16. Classic solution: Prod server(s) Kibana server(s) Our Service Log file(s) LogStash ElasticSearch Indexes Kibana How it works
- 17. LogStash: real-life example (live demo) ● Log file of Spring Boot application ● Log4j2 with MDC for entity grouping ● Sample GrOk config file (input, filter, output) ○ Discussing config file ○ Example of RegExing ○ Example of debugging GrOk matcher ● Start Elastic Stack on Windows machine ● Scan sample log-files ● Magic of Kibana ○ Discovery ○ Visualisation ○ Dashboard
- 18. How can we avoid pitfalls of LogStash? ● Takes 1GB or RAM (Java!) ⇒ Rewrite it on Go ● Parsing and computing @ onboard machine ⇒ Move to other machine ● Network consumption is tens of MBps ⇒ Zip data (?) ● Looooong start (Java!) ⇒ Rewrite it on Go ● GrOk and RegExp ⇒ Life is pain... ● Not easy to debug the script ⇒ Keep calm and heavy breathing...
- 19. Prod server(s) Kibana server(s) Our Service Log file(s) LogStash ElasticSearch Indexes Kibana FileBeat Beat solution: How it works now
- 20. ElasticBeats or “What else do you want from me?” Beats (Lightweight Data Shippers) is the platform for single-purpose data shippers. They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch. https://www.elastic.co/products/beats ● Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Heartbeat, etc… ● Tens of community beats... ● … and custom Beat constructor
- 21. Useful links ● About https://en.wikipedia.org/wiki/Kibana ● https://www.elastic.co/products/kibana ● https://www.elastic.co/products/logstash ● https://www.elastic.co/products/elasticsearch ● Easy install-config manual: http://knes1.github.io/blog/2015/2015-08-16-manage-spring-boot-logs-with- elasticsearch-kibana-and-logstash.html ● Default patterns for Grok parser: https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns ● Web-site to test regular expressions: https://regex101.com/ ● Web-site to test Grok expression: http://grokconstructor.appspot.com ● Search how-to: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/query-dsl-query-string- query.html#query-string-syntax
- 22. Thank you for your attention! Any Questions?