DevOpsDays Amsterdam 2016 workshop
- 1. DevOpsDays 2016 Amsterdam First steps to create your IT Operations data lake with ELK Stack and Graphite
- 2. 2 Arnold van Wijnbergen @BSMConsultant / Arnold.van.Wijnbergen@gmail.com •16 years working experience with automating IT •DevOps Evangelist with great passion about Architecture and Automation Tooling •Full time Tooling Geek Automation Monitoring, tasted them all Predictive Analytics Infrastructure-as-Code #IlovIT ☁
- 3. 3 INTRODUCE OURSELVES Who we are and what do we expect
- 4. 4 CHOOSE YOUR WAY Dev, Ops or just DevOps DOD-AMS-WORKSHOP-BANKIT TOURIST ROUTE DOD-AMS-WORKSHOP DANGER ROUTE
- 5. JUST ANOTHER DATA LAKE CONCEPT Metrics, logs, health-states, etc
- 6. “ELK Stack + Graphite + Kafka makes a great combination”
- 8. Main component for managing your element is LogStash Reference : https://www.elastic.co/guide/en/logstash/current/introduction.html
- 9. Logstash Structured way of Working – Processing Pipeline Reference : https://www.elastic.co/guide/en/logstash/current/pipeline.html
- 10. Pipeline processing of an event is mainly existing of three important stages Reference : https://www.elastic.co/guide/en/logstash/current/filter-plugins.html (Filter) Processing Parse fields out Enrich fields values Correlate On field value matching
- 11. Parse unstructured data and make IT readable structured
- 12. Enrich & correlate our structured data set
- 13. Success with the workshop https://goo.gl/97xOX2 13
- 14. Backup slides 14
- 15. Meet Mr LogStash What can he do for you ? I Love your data … :-)
- 16. Many test options are available to validate your code and expected functional behaviour Code Validation with CONFIGTEST Unit Testing with RSPEC Reference : https://www.elastic.co/blog/logstash-functionality-through-testing
- 17. Now combine the forces and create a structured configuration Configuration folder Input.conf Input.conf Input.conf Filter.conf Input.conf Output.conf file stdi n mutat egro k stdou telasticsear chredi s Reference : https://www.elastic.co/guide/en/logstash/current/configuration-file- structure.html
Editor's Notes
- #3: Goal of this presentation is that with infra monitoring your are not there yet
- #17: Syntax validation logstash –configtest Functional testinglogstash.rb logstash –rspec # encoding: utf-8 require "test_utils" file = "/etc/logstash/conf.d/test.conf" @@configuration = String.new @@configuration << File.read(file) describe "Test event" do extend LogStash::RSpec config(@@configuration) message = %(2015-21-01 12:01:02.003 UTC Hello 42 3.14159) sample("message" => message, "type" => "test") do insist { subject["type"] } == "test" insist { subject["timestamp"] } == "2015-21-01 12:01:02.003 UTC" insist { subject["word1"] } == "Hello" insist { subject["int1"] } == 42 insist { subject["inner"]["float1"] } == 3.14159 end end