Unifying Events and Logs into the Cloud
0 likes498 views
1. Eduardo Silva discussed unifying event and log data from multiple sources into the cloud using Fluentd and Fluent Bit. 2. Fluentd is an open source data collector that allows for parsing and storing data from multiple sources through its pluggable input and output plugins. 3. Fluent Bit is designed for collecting data from IoT and embedded devices to transport it to third party services, with a focus on performance and lightweight resource usage.
Unifying Events and Logs into the Cloud
- 1. Eduardo Silva eduardo@treasuredata.com @edsiper Unifying Events & Logs into the Cloud October 06, 2015 StartechConf, Santiago
- 2. About Me Eduardo Silva ● Github & Twitter @edsiper ● Personal Blog http://edsiper.linuxchile.cl Treasure Data ● Open Source Engineer ● Fluentd / Fluent Bit http://github.com/fluent Projects ● Monkey HTTP Server http://monkey-project.com ● Duda I/O http://duda.io
- 3. Logging
- 4. Logging Matters Pros ● Application status ● Debugging ● General information about anomalies: errors ● Troubleshooting / Support ● Local or Remote (network)
- 5. Logging Matters From a business point of view ● Input data Analytics→ ● User interaction / behaviors ● Improvements
- 6. Assumptions
- 7. Logging Matters Assumptions ● I have enough disk space ● I/O operations will not block ● Log messages are human readable ● My logging mechanism scale
- 8. Logging Matters Assumptions Basically, yeah.. it should work.
- 9. Concerns
- 10. Logging Matters Concerns ● Logs increase = data increase ● Message format get more complex ● Did the Kernel flush the buffers ? (sync(2)) ● Multi-thread application ?, locking ? ● Multiple Applications = Multiple Logs
- 11. Logging Matters Concerns If Multiple Applications = Multiple logs Multiple Hosts x Multiple Applications = ???
- 12. OK, so: 1. Logging matters 2. It's really beneficial 3. but...
- 13. It needs to be done right.
- 14. Logging Common sources & inputs ● Application Logs ● Apache ● NginX ● Syslog (-ng) ● Custom applications / Languages ● C, Ruby, Python, PHP, Perl, NodeJS, Java, etc.
- 15. In a galaxy not so far away...
- 17. How to parse/store multiple data sources ? note: performance matters!
- 19. Fluentd is an open source data collector It let's you unify the data collection for a better use and understanding of data.
- 20. before
- 21. after
- 22. Fluentd Highlights ● High Performance ● Built-in Reliability ● Structured Logs ● Pluggable Architecture ● More than 300 plugins! (input/filtering/output)
- 30. M x N M + N→
- 32. Fluentd Simple Forwarding: configuration # logs from a file # store logs to MongoDB <source> <match backend.*> type tail type mongo path /var/log/httpd.log database fluent format apache2 collection test tag backend.apache </match> </source> # logs from client libraries <source> type forward port 24224 </source>
- 35. Fluentd # logs from a file # store logs to MongoDB <source> <match *.*> type tail type copy path /var/log/httpd.log <store> format apache2 type elasticsearch tag backend.apache logstash_format true </source> </store> # logs from client libraries <store> <source> type webhdfs type forward host 192.x.y.z port 24224 port 50070 </source> path /path/to/hdfs </store> </match>
- 36. Who uses Fluentd in production ?
- 39. We collect 1M events per second !
- 41. Internet of Things Facts ● IoT will grow to many billions of devices over the next decade. ● Now it's about device to device connectivity. ● Different frameworks and protocols are emerging. ● It needs Logging.
- 42. Internet of Things Alliances Vendors formed alliances to join forces and develop generic software layers for their products:
- 43. Internet of Things Solutions provided Alliance Framework → →
- 44. IoT & Big Data Analytics IoT requires a generic solution to collect events and data from different sources for further analysis. Data can come from a specific framework, radio device, sensor or other. How do we collect and unify data properly ?
- 45. @fluentbit
- 46. Fluent Bit is an open source data collector It let's you collect data from IoT/Embedded devices and transport It to third party services.
- 47. Fluent Bit Targets ● Services ● Sensors / Signals / Radios ● Operating System information ● Automotive / Telematics
- 48. Fluent Bit Requirements IoT and Embedded environment requires special handling, specifically on performance and resource utilization: ● Lightweight ● Written in C Language ● Customizable, pluggable architecture ● Full integration with Fluentd
- 51. Containers
- 53. Docker Logging driver ● Docker v1.6 released the concept of logging drivers ● Route container output ● Fluentd ?
- 54. Docker
- 55. Docker v1.8 Fluentd Logging driver!
- 60. We Love Data! Thank you! ● http://fluentd.org ● http://fluentbit.io ● https://docs.docker.com/reference/logging/fluentd/ ● http://github.com/fluent/fluentd