Fluentd at HKOScon
0 likes1,437 views
Fluentd is a data collector for unified logging that provides a robust core and plugins. It allows for reliable data transfer through error handling and retries. The core handles common concerns like parsing, buffering, and writing data, while plugins handle input, output, and other use cases. Fluentd has a pluggable architecture and processes data through a pipeline of input, parser, filter, buffer, formatter, and output plugins.
![> logged time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> Actual content
> JSON / MessagePack
✓ Record
127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...
...
2012-12-11 07:26:27
apache.log
{
"host": "127.0.0.1",
"method": "GET",
...
}
Apache log Fluentd event
convert
time
tag
record](https://image.slidesharecdn.com/fluentd-hkoscon-160630030659/85/Fluentd-at-HKOScon-11-320.jpg)
Fluentd at HKOScon
- 1. Fluentd: Unified logging layer June 25, 2016 / HKOScon Masahiro Nakagawa
- 2. Who am I • Masahiro Nakagawa • github: @repeatedly • Treasure Data, Inc. • Senior Software Engineer • Fluentd / td-agent developer • Living at OSS :) • D language - Phobos, a.k.a standard library, committer • Fluentd - Main maintainer • MessagePack / RPC • etc…
- 3. Background
- 4. Before ✓ duplicated code for error handling... ✓ messy code for retrying mechanism...
- 5. After
- 6. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
- 8. What’s Fluentd? • Data collector for unified logging layer • Streaming data transfer based on JSON / MessagePack • Robust core and plugins written in Ruby • Rubygems based various plugins by the community • www.fluentd.org/plugins • Apache License, Version 2.0 • https://github.com/fluent/fluentd
- 9. Reliable data transfer error retry error retry retry retry Batch Stream Other stream (micro-batch)
- 10. Core Common concerns Use case specific PLUGINSCORE • Read Data • Parse Data • Buffer Data • Write Data • Format Data • Divide & Conquer • Buffering & Retries • Error Handling • Message Routing • Parallelism
- 11. > logged time Event structure(log message) ✓ Time > for message routing ✓ Tag > Actual content > JSON / MessagePack ✓ Record 127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ... ... 2012-12-11 07:26:27 apache.log { "host": "127.0.0.1", "method": "GET", ... } Apache log Fluentd event convert time tag record
- 12. Architecture
- 13. Data processing pipeline ParserInput Buffer Output FormatterFilter “output-ish”“input-ish”
- 14. Input plugins File tail (in_tail) Syslog (in_syslog) HTTP (in_http) RDBMS (in_sql) ... ✓ Receive logs ✓ Or pull logs from data sources ✓ non-blocking InputInput
- 15. Parser plugins JSON Regexp Apache/Nginx Syslog CSV/TSV etc. ✓ Parse into JSON ✓ Common formats out of the box ✓ Some inputs plugin depends on Parser plugin ParseParser
- 16. Filter plugins grep record_transformer suppress … ✓ Filter / Mutate record ✓ Record level and Stream level ✓ v0.12 and above Filter
- 17. Buffer plugins ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safetyMemory (buf_memory) File (buf_file) Buffer
- 18. Buffer internal ✓ Chunk = adjustable unit of data ✓ Buffer = Queue of chunks chunk chunk chunk output Input
- 19. Formatter plugins ✓ Format output ✓ Convert json into other format ✓ Some plugins depends on Formatter plugins JSON CSV/TSV “single value” msgpack Formatter
- 20. Output plugins ✓ Write to external systems ✓ Buffered & Non-buffered ✓ 300+ pluginsFile (out_file) Amazon S3 (out_s3) Kafka (out_kafka) ... Output
- 21. Use cases
- 23. # logs from a file <source> @type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag backend.apache </source> # logs from client libraries <source> @type forward port 24224 </source> # store logs to MongoDB <match backend.*> @type mongo database fluent collection test </match>
- 24. Less Simple Forwarding - At-most-once / At-least-once - HA (failover) - Load-balancing
- 26. Treasure Data Frontend Job Queue Worker Hadoop Presto Fluentd Applications push metrics to Fluentd (via local Fluentd) Datadog for realtime monitoring / alerting Treasure Data for historical analysis Fluentd sums up data minutes (partial aggregation)
- 29. Roadmap
- 30. History / Roadmap • v0.10.0 (In Oct 2011) • v0.12.0 (In Dec 2014) -> Current stable • v0.14.0 (In May 2016) • v0.14.x (some versions in 2016) • v1 (4Q in 2016 / 1Q in 2017)
- 31. v0.10 (old stable) > First stable release > Mainly for log forwarding > Only Input and Output > No complex event handling support > Only At-most-once semantics in forwarding > Treasure Data provides td-agent 1 for v0.10
- 32. v0.12 (current stable) > Event handling improvement > Filter, Label, Error Stream > New configuration format > Add at-least-once semantics in forwarding > Use require_ack_response parameter > HTTP RPC based process management > Treasure Data provides td-agent 2 for v0.12
- 33. • New Plugin APIs, Plugin Helpers & Plugin Storage • Supervisor using ServerEngine • Time with Nanosecond precision • Windows support v0.14 (Next stable)
- 34. Time with nanosecond • Fluent::EventTime • behaves as Integer (used as time in v0.12) • has methods to get sub-second precision • be serialized into msgpack using Ext type • Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco- system (e.g., fluent-logger, Docker logging driver)
- 35. Windows support • Fluentd and core plugin work on Windows • several companies have already used v0.14.0.pre version on production • We will send a patch to popular plugins if it doesn’t work on Windows • Use HTTP RPC instead of signals • Treasure Data will provide td-agent msi package
- 36. v0.14.x - v1 • v0.14.x (some versions in 2016) • Symmetric multi processing model • Counter API • TLS/authentication/authorization support (merging secure-forward into core) • v1 (4Q in 2016 / 1Q in 2017) • Stable version for new APIs/features • fully compatible with v0.12.x and v0.14.x
- 37. Symmetric multi processing model • 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine) • Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one (or some) output plugin • No more fluent-plugin-multiprocess
- 38. Worker Supervisor Worker Worker Worker Supervisor Worker Worker Supervisor Supervisor Using fluent-plugin-multiprocess v0.14
- 39. TLS/Authn/Authz support for forward plugin • secure-forward will be merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/ non-SSL forwarding • Authentication and Authorization providers • Who can connect to input plugins? What tags are permitted for clients? • New plugin types (3rd party authors can write it) • Mainly for in/out forward, but available from others
- 40. Ecosystem
- 41. Treasure Agent (td-agent) > Treasure Data distribution of Fluentd > include Ruby runtime, fluentd and popular plugins > deb, rpm, dmg are supported > Treasure Agent 2 is current stable > fluentd v0.12 and Ruby 2.1 > Treasure Agent 3 will be released at fall > fluentd v0.14 and Ruby 2.3 > msi will be supported for Windows
- 42. fluentd-ui > Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui
- 43. fluent-bit > http://fluentbit.io/ > Log collector mainly for embedded systems > Written in C > Can communicate with Fluentd > There are several plugins > Treasure Data, Elasticsearch, MQTT, etc…
- 44. > Bulk Loader version of Fluentd > Pluggable architecture > JRuby, JVM languages > High throughput by parallel processing > Invented by Treasure Data > Share your script as a plugin > https://github.com/embulk Slide: Fighting Against Chaotically SeparatedValues with Embulk