Open source data ingestion
6 likes4,941 views
This document discusses data collection and ingestion tools. It begins with an overview of data collection versus ingestion, with collection happening at the source and ingestion receiving the data. Examples of data collection tools include rsyslog, Scribe, Flume, Logstash, Heka, and Fluentd. Examples of ingestion tools include RabbitMQ, Kafka, and Fluentd. The document concludes with a case study of asynchronous application logging and challenges to consider.
Open source data ingestion
- 1. Open Source Data Collection/Ingestion Treasure Data, Inc. www.treasuredata.com
- 2. Hello! - “Committer” of Fluentd - Treasure Data, Inc. - Former Algorithmic Trader - Stanford Math and CS
- 3. Table of Contents 1. Why you should care 2. Data Collection v. Data Ingestion 3. Examples: Data Collection Tools 4. Examples: Data Ingestion Tools 5. Case Study: Async App Logging Links to be added after the talk.
- 4. Data Collection/Ingestion is HARD
- 5. Data Sources Raw Data Storage Processed Data Analysis Environment (Big) Data Pipeline Data Collection and Ingestion Data Pre- processing Data Fetching Data Engineers
- 6. Data Sources Raw Data Storage Processed Data Analysis Environment If Data Collection Goes Awry... Data Collection and Ingestion Data Pre- processing Data Fetching Data Engineers
- 8. Data Collection - Happens where data originates - “logging code” - Batch v. Streaming - Pull v. Push log.error(“FUUUUU....WHY!?”) cln.send({“uid”:1,”action”:”died”}) 200 GET a.com/?utm=big%20data
- 9. Data Ingestion - Receives data - Sometimes coupled with storage - Routing data Data Ingestion Layer
- 10. ex. Data Collection Tools
- 11. rsyslog - The grandfather of data collectors - Streaming - Installed by default, widely understood - Not as easy to extend/configure
- 13. Scribe - Written originally at Facebook - Streaming - Fast (C++) - Nightmare to build, largely abandoned
- 14. Flume-ng - Written and maintained by Cloudera (successor to Flume) - Commercial support by Cloudera. Track record for Hadoop - Java can be heavy-handed for some orgs/cases
- 15. Logstash - Pluggable architecture, rich ecosystem - The “L” of the ELK stack by Elastic - JRuby - HA uses Redis as a queue http://apuntesdetrabajo.es/?p=263
- 16. Heka - Developed at Mozilla - Written in Go, extensible w/ Lua - Plugin system, but compilation needed (Go’s limitation, may change)
- 17. Fluentd - Plugin architecture - Built-in HA - CRuby (JRuby on the roadmap) - google-fluentd, td-agent - Lightweight multi-source, multi- destination log routing
- 18. Embulk - Plugin architecture - Focuses on Batch workloads - Java/JRuby - Very new! (looking for contributors!)
- 19. ex. Data Ingestion Tools
- 20. RabbitMQ - Written in Erlang, supported by Pivotal - Implements AMQP
- 21. Kafka - Begun at LinkedIn, now Confluent - Topic-based Message Broker: Producer/Broker/Consumer - Distributed design - Provides at least once, at most once by consumers
- 22. Fluentd!? - Used (abused?) as a bus/MQ - tag-based event routing - Can be combined with RabbitMQ/Kafka, etc.
- 23. case study: Async App Logging
- 24. Application Logging - Common ask: “How’s our new feature doing?” GET /foobar API Server 200 {...}
- 25. Application Logging - What NOT to do: synchronous logging GET /foobar API Server200 {...} Data Backend write ack
- 26. Application Logging - What NOT to do: synchronous logging GET /foobar API Server200 {...} Local Data Collector write Flush Data Backendack Buffer
- 27. - Is writing to a local log collector safe? - What if the log collector retries by error? But wait... - A lot of problems to think about!
- 28. “Much of the blame, little of the glory” (Just kidding. The entire data team relies on YOU!)
- 29. Thank you! (...and we are hiring!) www.treasuredata.com/careers
- 30. - Software - www.fluentd.org - hekad.readthedocs.org - logstash.org - kafka.apache.org - Embulk.org - www.rabbitmq.com - Ideas - https://engineering.linkedin.com/distributed-systems/log-what-every- software-engineer-should-know-about-real-time-datas-unifying - http://radar.oreilly.com/2015/04/the-log-the-lifeblood-of-your-data- pipeline.htmlL Bibliography