Fluentd meetup in japan
67 likes82,404 views
Fluentd is an open source data collector designed for log aggregation and processing. It has a pluggable architecture that allows for flexible input, filtering, and output of log data. It is useful for log collection, processing, and forwarding to storage or analysis services in real-time.
![Plugins :: in_tail
Apache log parser
87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98
87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98
...
{
“host”: “87.12.1.87”,
“method”: “GET”,
“code”: 200,
“size”: 98,
“path”: “/”
}
...](https://image.slidesharecdn.com/fluentdmeetupinjapan-120203174115-phpapp01/85/Fluentd-meetup-in-japan-28-320.jpg)
Fluentd meetup in japan
- 1. Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector Service Sadayuki Furuhashi Treasure Data, Inc. @frsyuki
- 2. • Sadayuki Furuhashi > twitter: @frsyuki • Treasure Data, Inc. > Software Engineer; founder • Author of MessagePack • Author of Fluentd
- 3. What’s Fluentd? It's like syslogd, but uses JSON for log messages
- 4. What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
- 5. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
- 6. What’s Fluentd? Application Fluentd lter / bu er / routing Storage
- 7. What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
- 8. What’s Fluentd? syslogd Scribe Application File Plug-in tail Plug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
- 9. What’s Fluentd? • Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ... Fluent.open(“myapp”) Fluent.event(“login”, {“user”=>38}) #=> 2012-02-04 04:56:01 myapp.login {“user”:38}
- 10. Fluentd & Event logs Before: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
- 11. Fluentd & Event logs After: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
- 12. Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
- 13. # receive events via HTTP # save alerts to a file <source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts </source> </match> # read logs from a file # forward other logs to servers <source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11 </source> weight 20 </server> # save access logs to MongoDB <server> <match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server> </match> </match>
- 14. Fluentd vs Scribe • Deals with structured logs • Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/ • Easy to customize • add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
- 15. Fluentd vs Flume • Easy to setup > “sudo fluentd --setup && fluentd” • Very small footprint > small engine (3,000 lines) + plugins • JVM-free • Easy to configure
- 18. Architecture :: Bu er
- 20. Plugins :: out_forward forward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
- 21. Plugins :: out_forward forward event logs Fluentd Heartbeat out_forward φ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
- 22. Plugins :: out_copy duplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
- 23. Plugins :: buf_ le reliable bu ering buf_ le Fluentd le le ✓ Automatic retry le ✓ 2^N retry interval le ✓ Persistent bu er
- 24. Plugins :: out_exec execute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
- 25. Plugins :: out_exec_ lter execute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
- 26. Plugins :: in_exec execute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
- 27. Plugins :: in_tail Read event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
- 28. Plugins :: in_tail Apache log parser 87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98 87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98 ... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
- 29. Plugins • Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
- 30. Plugins • 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
- 31. Plugin developer API • Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver • Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
- 32. Fluentd • Documents > http://fluentd.org • Source code > http://github.com/fluent • Twitter > #fluentd • Mailing list > http://groups.google.com/group/fluentd