Fluentd - Unified logging layer
3 likes1,822 views
Fluentd is an open source data collector that allows for flexible and extensible logging. It provides a unified way to collect logs, metrics, and events from various sources and send them to multiple destinations. It handles concerns like buffering, retries, and failover to provide reliable data transfer. Fluentd uses a plugin-based architecture so it can support many use cases like simple forwarding, lambda architectures, stream processing, and logging for Docker and Kubernetes.
Fluentd - Unified logging layer
- 1. FLUENTD: UNIFIED LOGGING LAYER Kiyoto Tamura July 23, 2015 Silicon Valley Data Engineering Meetup
- 2. WHOAMI Kiyoto Tamura GitHub/Twitter: kiyoto/kiyototamura Treasure Data, Inc. Vice President, Marketing TRUST ME, I used to code daily =)
- 3. Tweet NOW! “At #svde learning how to collect more event data using #Fluentd”
- 4. WHAT’S FLUENTD? An extensible & reliable data collection tool simple core + plugins buffering, HA (failover), load balancing, etc. like syslogd
- 6. ✓ duplicated code for error handling... ✓ messy code for retrying mechnism... Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system bash scripts ruby scripts rsync log file bash python scripts custom loggger cron other custom scripts...
- 8. Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system filter / buffer / route
- 9. extensible
- 10. CORE PLUGINS • Divide & Conquer • Buffering & Retries • Error Handling • Message Routing • Parallelism • Read Data • Parse Data • Buffer Data • Write Data • Format Data Common Concerns Use Case Specific
- 11. architecture
- 12. INTERNAL ARCHITECTURE “input-ish” “output-ish” Input Parser Buffer Output FormatterFilter
- 13. reliable
- 15. DIVIDE & CONQUER & RETRY error retry error retry retry retry
- 16. reliable process
- 17. THIS?
- 18. OR THIS?
- 19. M X N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer/filter/route
- 20. use cases
- 22. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match backend.*> type mongo database fluent collection test </match>
- 25. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match backend.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
- 26. CEP FOR STREAM PROCESSING
- 27. FLUENTD ON KUBERNETES (NOV 2015)
- 28. FLUENTD LOGGING DRIVER (APR 2015)
- 29. Tweet Again! “Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver”
- 30. DEMO: FLUENTD + DOCKER
- 31. THANK YOU! AND TREASURE DATA IS HIRING! WWW.TREASUREDATA.COMC/CAREERS