![Flume Goals: Manageability
Configuring Flume
Node: tail(“file”) | filter [ console, roll
(1000) { dfs(“hdfs://namenode/user/flume”) } ]
;
Output Bucketing
/logs/web/2010/0715/1200/data-xxx.txt
/logs/web/2010/0715/1200/data-xxy.txt
/logs/web/2010/0715/1300/data-xxx.txt
/logs/web/2010/0715/1300/data-xxy.txt
/logs/web/2010/0715/1400/data-xxx.txt](https://image.slidesharecdn.com/flume-120314204418-phpapp01/85/Apache-Flume-15-320.jpg)
Apache Flume
- 1. Arinto Murdopo Josep Subirats Group 4 EEDC 2012
- 2. Outline ● Current problem ● What is Apache Flume? ● The Flume Model ○ Flows and Nodes ○ Agent, Processor and Collector Nodes ○ Data and Control Path ● Flume goals ○ Reliability ○ Scalability ○ Extensibility ○ Manageability ● Use case: Near Realtime Aggregator
- 3. Current Problem ● Situation: You have hundreds of services running in different servers that produce lots of large logs which should be analyzed altogether. You have Hadoop to process them. ● Problem: How do I send all my logs to a place that has Hadoop? I need a reliable, scalable, extensible and manageable way to do it!
- 4. What is Apache Flume? ● It is a distributed data collection service that gets flows of data (like logs) from their source and aggregates them to where they have to be processed. ● Goals: reliability, scalability, extensibility, manageability. Exactly what I needed!
- 5. The Flume Model: Flows and Nodes ● A flow corresponds to a type of data source (server logs, machine monitoring metrics...). ● Flows are comprised of nodes chained together (see slide 7).
- 6. The Flume Model: Flows and Nodes ● In a Node, data come in through a source... ...are optionally processed by one or more decorators... ...and then are transmitted out via a sink. Examples: Console, Exec, Syslog, IRC, Twitter, other nodes... Examples: Console, local files, HDFS, S3, other nodes... Examples: wire batching, compression, sampling, projection, extraction...
- 7. The Flume Model: Agent, Processor and Collector Nodes ● Agent: receives data from an application. ● Processor (optional): intermediate processing. ● Collector: write data to permanent storage.
- 8. The Flume Model: Data and Control Path (1/2) Nodes are in the data path.
- 9. The Flume Model: Data and Control Path (2/2) Masters are in the control path. ● Centralized point of configuration. Multiple: ZK. ● Specify sources, sinks and control data flows.
- 10. Flume Goals: Reliability Tunable Failure Recovery Modes ● Best Effort ● Store on Failure and Retry ● End to End Reliability
- 11. Flume Goals: Scalability Horizontally Scalable Data Path Load Balancing
- 12. Flume Goals: Scalability Horizontally Scalable Control Path
- 13. Flume Goals: Extensibility ● Simple Source and Sink API ○ Event streaming and composition of simple operation ● Plug in Architecture ○ Add your own sources, sinks, decorators
- 14. Flume Goals: Manageability Centralized Data Flow Management Interface
- 15. Flume Goals: Manageability Configuring Flume Node: tail(“file”) | filter [ console, roll (1000) { dfs(“hdfs://namenode/user/flume”) } ] ; Output Bucketing /logs/web/2010/0715/1200/data-xxx.txt /logs/web/2010/0715/1200/data-xxy.txt /logs/web/2010/0715/1300/data-xxx.txt /logs/web/2010/0715/1300/data-xxy.txt /logs/web/2010/0715/1400/data-xxx.txt
- 16. Use Case: Near Realtime Aggregator
- 17. Conclusion Flume is ● Distributed data collection service ● Suitable for enterprise setting ● Large amount of log data to process
- 18. Q&A Questions to be unveiled?
- 19. References ● http://www.cloudera. com/resource/chicago_data_summit_flume_an_introduction_jonathan_hsie h_hadoop_log_processing/ ● http://www.slideshare.net/cloudera/inside-flume ● http://www.slideshare.net/cloudera/flume-intro100715 ● http://www.slideshare.net/cloudera/flume-austin-hug-21711