OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
0 likes139 views
The document discusses recent developments in Logstash, including enhancements in log ingestion and monitoring capabilities. Notable changes include a transition to Filebeat as the new log harvester, updates to configuration management allowing for dynamic reloading of configs, and the introduction of native offline plugin management. Additionally, it outlines planned changes for future releases, including a new event API and breaking changes that may affect existing implementations.
![Configuring Filebeat
Outputs
24
################### Filebeat Configuration Example #########################
output:
logstash:
hosts: ["localhost:5044"]
index: filebeat
……
elasticsearch:
hosts: ["localhost:9200"]
……](https://image.slidesharecdn.com/pereurbon-bayes-ingestinglogswithstyle-160502102430/85/OSDC-2016-Ingesting-Logs-with-Style-by-Pere-Urbon-Bayes-24-320.jpg)
![This that might break up for 5.0
• New Event API proposal [#5141]
• bin/plugin is now bin/logstash-plugin [#4871]
• Require Java 8 as default [#3877]
• Fieldref StringInterpolation and Fixnum/Float values [#5114]
• Add support for x-pack’s [#4872]
• Runner cleanup [#3148, #3860]
• Use settings file to complement command line arguments [#4401]
30](https://image.slidesharecdn.com/pereurbon-bayes-ingestinglogswithstyle-160502102430/85/OSDC-2016-Ingesting-Logs-with-Style-by-Pere-Urbon-Bayes-30-320.jpg)
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
- 1. ‹#› What has been cooking lately in Logstash world. Ingesting Logs with style
- 2. $whoami Pere Urbon-Bayes (Software Engineer since ever) Have always worked with databases, data and analytics. GraphDevRoom@FOSDEM When not coding I enjoy my time with my wife and kid. I also enjoy movies and tv series, and used to like running, too. Basically, I’m doing everything to enjoy live. 2
- 3. 3 Life demo?
- 5. 5 Metrics
- 6. What is the current Logstash throughput? 6 Logstash is blocked?, ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……
- 7. Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 7
- 8. 8
- 11. Step 1: Config Reloading Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 11 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?
- 12. 12 role: frontend-logs 1 Create a role, upload config
- 14. The java reincarnation • Logstash is based on JRuby, Ruby on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 14
- 17. 17
- 18. Native offline plugins management support • Build your plugins bundle to be used in offline environments • bin/logstash-plugin pack • bin/logstash-plugin unpack • bin/logstash-plugin install --local • Use your own private plugins server • Geminabox, Gemirro, Artifactory, Gemfury, … • Edit the Gemfile to use the new server. Manage plugins in offline environments made easier 18
- 19. 19 Pipeline, the next generation
- 20. The pipeline components 20 I F O I F O
- 21. 21 Filebeat, the new log harvester gen
- 22. Long life to Logstash Forwarder, welcome Filebeat • LSF is deprecated, next generation harvester is Filebeat. • Written in Go and configured in YAML. • Available as deb, rpm, linux (32 and 64bit), mac and windows binaries. • Similar feature sets as LSF, including: • Send data direct to Elasticsearch • Send data to Logstash (logstah-forwarder protocol 2.0) • Security as first class citizen • HTTPS for Elasticsearch • TLS for Logstash 22
- 23. Configuring Filebeat Prospectors 23 ################### Filebeat Configuration Example ######################### filebeat: prospectors: - paths: - /Users/purbon/lib/moves/*.tcx input_type: log multiline: pattern: ^<?xml negate: true match: after max_lines: 55000 timeout: 5s
- 24. Configuring Filebeat Outputs 24 ################### Filebeat Configuration Example ######################### output: logstash: hosts: ["localhost:5044"] index: filebeat …… elasticsearch: hosts: ["localhost:9200"] ……
- 25. Migration to Filebeat • Configuration file changed from JSON to YAML • The registry file has been changed, this file track the data transfer status • Recommendation: Use both in parallel until data sync is complete • Command line options where moved as settings • Mostly all removed, filebeat provides default values in settings. • The new beats input is required in the Logstash side. • Latest version enable LSF to talk to the beats input plugins, reducing migration friction. 25
- 27. ‹#›27 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson Magnus Bäck Philippe WeberJurgens du Toit Russell Savage
- 30. This that might break up for 5.0 • New Event API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 30
- 31. 31 Demo time