The basics of fluentd
56 likes40,725 views
Fluentd is an open source data collector that allows flexible data collection, processing, and storage. It collects log data from various sources using input plugins and sends the data to various outputs like files, databases or forward to other Fluentd servers. It uses a pluggable architecture so new input/output plugins can be added through Ruby gems. It provides features like buffering, retries and reliability.
![Time 2012-12-11 07:26:27
Apache Tag apache.log
Record {
"host": "127.0.0.1",
tail "method": "GET",
"path": "/",
write ...
}
insert
127.0.0.1
127.0.0.1
127.0.0.1
-
-
-
-
-
-
[11/Dec/2012:07:26:27]
[11/Dec/2012:07:26:30]
[11/Dec/2012:07:26:32]
"GET
"GET
"GET
/
/
/
...
...
...
Fluentd
127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:27:01] "GET / ...
...
event
buffering
Mongo](https://image.slidesharecdn.com/the-basics-of-fluentd-130307051548-phpapp01/85/The-basics-of-fluentd-12-320.jpg)
The basics of fluentd
- 1. The basics of Fluentd Masahiro Nakagawa Treasuare Data, Inc. Senior Software Engineer
- 2. Structured logging Reliable forwarding http://fluentd.org/ Pluggable architecture
- 3. Agenda > Background > Overview > Product Comparison > Use cases
- 4. Background
- 5. Data Processing Data source Collect Store Process Visualize Reporting Monitoring
- 6. Related Products easier & shorter time Collect Store Process Visualize ??? Cloudera Excel Horton Works Tableau Treasure Data R
- 8. Before Fluentd Server1 Server2 Server3 Application Application Application ･･･ ･･･ ･･･ High Latency! must wait for a day... Fluent Log Server
- 9. After Fluentd Server1 Server2 Server3 Application Application Application Fluentd ･･･ Fluentd ･･･ Fluentd ･･･ In streaming! Fluentd Fluentd
- 10. Overview
- 11. In short > Open sourced log collector written in Ruby > Using rubygems ecosystem for plugins It’s like syslogd, but uses JSON for log messages
- 12. Time 2012-12-11 07:26:27 Apache Tag apache.log Record { "host": "127.0.0.1", tail "method": "GET", "path": "/", write ... } insert 127.0.0.1 127.0.0.1 127.0.0.1 - - - - - - [11/Dec/2012:07:26:27] [11/Dec/2012:07:26:30] [11/Dec/2012:07:26:32] "GET "GET "GET / / / ... ... ... Fluentd 127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:27:01] "GET / ... ... event buffering Mongo
- 13. Event structure(log message) ✓ Time ✓ Record > second unit > JSON format > from data source or > MessagePack adding parsed time internally ✓ Tag > non-unstructured > for message routing
- 14. Architecture Pluggable Pluggable Pluggable Input Buffer Output > Forward > Memory > Forward > HTTP > File > File > File tail > Amazon S3 > dstat > MongoDB > ... > ...
- 15. Client libraries > Ruby > Java Application > Perl > PHP Time:Tag:Record > Python >D > Scala > ... Fluentd # Ruby Fluent.open(“myapp”) Fluent.event(“login”, {“user” => 38}) #=> 2012-12-11 07:56:01 myapp.login {“user”:38}
- 16. Configuration and operation > No central / master node > HTTP include helps conf sharing > Operation depends on your environment > Use your deamon management > Use chef in Treasure Data > Scribe like syntax
- 17. # receive events via HTTP # save alerts to a file <source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts </source> </match> # read logs from a file # forward other logs to servers <source> <match **> type tail type forward path /var/log/httpd.log <server> format apache host 192.168.0.11 tag apache.access weight 20 </source> </server> <server> # save access logs to MongoDB host 192.168.0.12 <match apache.access> weight 60 type mongo </server> database apache </match> collection log </match> include http://example.com/conf
- 18. Reliability (core + plugin) > Buffering > Use file buffer for persistent data > buffer chunk has ID for idempotent > Retrying > Error handling > transaction, failover, etc on forward plugin > secondary
- 19. Plugins - use rubygems $ fluent-gem search -rd fluent-plugin $ fluent-gem search -rd fluent-mixin $ fluent-gem install fluent-plugin-mongo ※ Today, don’t talk the plugin development
- 21. in_tail Apache Fluentd ✓ read a log file ✓ custom regexp ✓ custom parser in Ruby access.log Supported format: > apache > json > apache2 > csv > syslog > tsv > nginx > ltsv (since v0.10.32)
- 22. out_mongo Apache Fluentd access.log buffer ✓ retry automatically ✓ exponential retry wait ✓ persistent on a file
- 23. out_webhdfs ✓ custom text formatter Apache Fluentd access.log buffer HDFS ✓ slice files based on time ✓ retry automatically 2013-01-01/01/access.log.gz ✓ exponential retry wait 2013-01-01/02/access.log.gz ✓ persistent on a file 2013-01-01/03/access.log.gz ...
- 24. out_copy + other plugins Hadoop Apache Fluentd access.log buffer Amazon S3 ✓ routing based on tags ✓ copy to multiple storages
- 25. out_forward ✓ automatic fail-over ✓ load balancing Fluentd apache Apache Fluentd Fluentd Fluentd access.log buffer ✓ retry automatically ✓ exponential retry wait ✓ persistent on a file
- 26. Forward topology Fluentd send/ack Fluentd Fluentd send/ack Fluentd Fluentd Fluentd Fluentd
- 27. Access logs Alerting Apache Nagios App logs Analysis Frontend MongoDB Backend MySQL System logs Hadoop syslogd filter / buffer / routing Archiving Databases Amazon S3
- 28. Access logs Alerting Apache Nagios App logs Analysis Frontend MongoDB Backend MySQL System logs Hadoop syslogd filter / buffer / routing Archiving Databases Amazon S3
- 29. Access logs Alerting Apache Nagios App logs Analysis Frontend MongoDB Backend MySQL System logs Hadoop syslogd filter / buffer / routing Archiving Databases Amazon S3
- 30. td-agent > Open sourced distribution package of fluentd > ETL part of Treasure Data > Including useful components > ruby, jemalloc, fluentd > 3rd party gems: td, mongo, webhdfs, etc... td plugin is for TD > http://packages.treasure-data.com/
- 31. v11 > Breaking source code compatibility > Not protocol. > Windows support > Error handling enhancement > Better DSL configuration > etc: https://gist.github.com/frsyuki/2638703
- 33. Scribe Scribe: log collector by Facebook
- 34. Pros and Cons > ● Pros > Fast (written in C++) > ● Cons > Hard to install and extend Are you a C++ magician? > Deal with unstructured logs > No longer maintained Replaced with Calligraphus at Facebook
- 35. Flume Flume: distributed log collector by Cloudera Phisical Flume Master Topology Flume Flume Flume Logical Topology Hadoop HDFS
- 36. Network topology Master Agent ack Agent Collector Flume OG Collector Agent Collector send Agent Master Option Agent send/ack Agent Collector Flume NG Collector Agent Collector Agent
- 37. Pros and Cons > ● Pros > Using central master to manage all nodes > ● Cons > Java culture (Pros for Java-er?) Difficult configuration and setup > Difficult topology > Mainly for Hadoop less plugins?
- 38. Use cases
- 39. Treasure Data Frontend Worker Job Queue Hadoop Hadoop Applications push metrics to Fluentd sums up data minutes (via local Fluentd) Fluentd Fluentd (partial aggregation) Treasure Librato Data Metrics for historical analysis for realtime analysis
- 40. Cookpad hundreds of app servers Rails app td-agent sends event logs Daily/Hourly Google Batch Spreadsheet Rails app td-agent Treasure Data sends event logs MySQL Rails app td-agent Logs are available sends event logs after several mins. KPI Feedback rankings visualization Unlimited scalability Flexible schema Realtime Less performance impact ✓ Over 100 RoR servers (2012/2/4)
- 41. NHN Japan Archive Storage Web Servers Fluentd (scribed) Cluster Notifications STREAM (IRC) Fluentd Watchers Graph Tools webhdfs SCHEDULED BATCH BATCH ✓ 16 nodes Hadoop Cluster hive server ✓ 120,000+ lines/sec CDH4 Shib ShibUI ✓ 400Mbps at peak Huahin ✓ 1.5+ TB/day (raw) (HDFS, YARN) Manager http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013 by @tagomoris
- 42. Other companies
- 43. Conclusion > Fluentd is a widely-used log collector > There are many use cases > Many contributors and plugins > Keep it simple > Easy to integrate your environment