Pivotal Open Source: Using Fluentd to gain insights into your logs
Download as PPTX, PDF1 like737 views
This document discusses the use of Fluentd as a unified logging layer. Fluentd can parse log data into a common format, decouple log collection from data sources, and provide reliability, scalability and extensibility as logs are ingested from various systems and sent to different outputs like search, databases and storage. It provides an example of using Fluentd to parse Apache web server logs and tag them with metadata before outputting. The document advocates that Fluentd can help bring together logs from multiple systems and services to support common needs like analytics, monitoring and incident response.
![outage.needs? :logs
=> [“no”, “shit”]](https://image.slidesharecdn.com/cfmeetupsep2014-140922110509-phpapp02/85/Pivotal-Open-Source-Using-Fluentd-to-gain-insights-into-your-logs-16-320.jpg)
![127.0.0.1 - - [05/Feb/2012:17:11:55
+0000] "GET / HTTP/1.1" 200 140 "-"
"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko)
Chrome/18.0.1025.5 Safari/535.19"](https://image.slidesharecdn.com/cfmeetupsep2014-140922110509-phpapp02/85/Pivotal-Open-Source-Using-Fluentd-to-gain-insights-into-your-logs-28-320.jpg)
![[“05/Feb/2012:17:11:55”,“web.access”,{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}]
timestamp tag
record](https://image.slidesharecdn.com/cfmeetupsep2014-140922110509-phpapp02/85/Pivotal-Open-Source-Using-Fluentd-to-gain-insights-into-your-logs-33-320.jpg)
Pivotal Open Source: Using Fluentd to gain insights into your logs
- 1. Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
- 3. • Fluentd maintainer & community manager • data nerd • math nerd • nerd whoami
- 5. this talk isn’t about Big Data
- 6. it’s about Log Data
- 8. The life of a data scientist (me in 2009)
- 10. Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue… Yell at the engineers Some columns are missing!? Run the script…DIVISION BY ZERO!!!
- 11. Hmm…
- 14. analytics.needs? :logs => true
- 16. outage.needs? :logs => [“no”, “shit”]
- 17. Metrics and Monitoring (hint: you need logs)
- 19. Ops VPs Engineers Managers PMs More PMs
- 21. How can we do better?
- 22. How to Unify Logging (1) Common Interface + Decoupling Mobile Web IoT Message Queue Search Backend Analytic DB Archival Storage Unified Logging Layer Parse into a common data format Decouple from data sources
- 23. How to Unify Logging (2) Reliability & Scalability Mobile Web IoT Message Queue Search Backend Analytic DB Archival Storage Unified Logging Layer Need persistence/buff ering Robust retries and recovery
- 24. How to Unify Logging (3) Extensibility ? Web IoT ? Search Backend Analytic DB Archival Storage Unified Logging Layer Adding a new in/output must be easy Same for filters
- 25. Fluentd can help us unify logging
- 26. how it works
- 28. 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
- 29. { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }
- 31. Parse as JSON!
- 32. ?
- 33. [“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }] timestamp tag record
- 34. ?
- 35. web.mongodb web.file web.hdfs web.s3 web.mysql
- 36. Demo: Bring Your Own A/B Testing
- 37. How A/B Testing Starts website <script>…</script> A/B Testing SaaS
- 38. How A/B Testing Evolves Android iOS <script>…</script> A/B Testing SaaS 1 website A/B Testing SaaS 1 A/B Testing SaaS 1 <script>…</script> A/B Testing SaaS 1 event.post()… <script>…</script> event.post()…
- 39. How A/B Testing Evolves Android iOS <script>…</script> A/B Testing SaaS 1 website A/B Testing SaaS 1 A/B Testing SaaS 1 <script>…</script> A/B Testing SaaS 1 event.post()… <script>…</script> event.post()… Looks Familiar?
- 40. Bring Your Own A/B Testing! Android website iOS A/B Testing SaaS 1 A/B Testing SaaS 2 Analytic DB Archival Storage
- 42. { “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd” }
Editor's Notes
- #2: Thanks the organizers, Pivotal, audience
- #5: so, I am a big fan of spoilers when it comes to tech talks. I think spoilers give the audience a much better idea of what to expect. so here it is.
- #6: phew. I just said that. no, this talk is definitely not about “big data” besides poking fun around the buzzword… people can’t seem to agree on what it is. I want to talk about something far more concrete
- #9: I worked as a quantitative analyst for three years
- #12: definitely not just data engineers’ problem started to think more deeply about why logging becomes haphazard. talked to hundreds of people at treasure data. eventually, I had a couple of observations.
- #14: and here is another observation
- #23: the first requirement is common interface between data inputs and outputs. why? common interface -> one data can be stored into multiple places with the same semantics. You don’t know if you stick to the same backend system. You probably will need to piece together information from multiple data sources.
- #24: data pipelines fail format changes volume spikes hardware/IaaS hiccups Scalability matters You need to be able to scale out the logging layer.
- #25: New data sources/outputs come up Need to be able to extend your system
- #26: So, here is a rather self-aggrandizing claim: Fluentd can be that unified logging layer. In the rest of the talk, I will show you how.
- #27: yes, it is about log data!