Data streaming-systems
- 1. Every ad. Every sales channel. Every screen. One platform. Building Distributed Data Streaming System AshishTadose Lead Software Engineer Big Data Analytics - PubMatic
- 2. Agenda • What is stream processing • Streaming architecture • Scalable Data Ingestion • RealTime Streaming Processing system 2
- 3. What is Streaming Process ? 3
- 4. In simple words, Streaming is… 4
- 5. Batch & Streaming processing Data Generator Ingestion Distributed File system Processing Data Store Batch processing Data Generator Ingestion Message Queue Processing Data Store Stream Data processing
- 6. Batch & Streaming processing 6 Data Generator Ingestion Message Queue Processing Data Store Stream Data processing Distributed File system Processing Data Store Batch processing
- 7. Batch & Streaming processing 7 Data Generator Ingestion Message Queue Processing Data Store Stream Data processing Distributed File system Processing Data Store Batch processing
- 10. Ingestion Ecosystem • Sources • Machine data • External stream & syslogs • Data Collection • Flume • Kafka • Kinesis • Confluent 10
- 11. Flume • Easier to setup • Rich set of in-build tools • No inherent support for data replication • Nodes works in isolation • Memory channel vs File Channel 11
- 12. Kinesis 12
- 13. Kafka 13 http://kafka.apache.org/ Originated at LinkedIn, open sourced in early 2011 Implemented in Scala, some Java 9 core committers, plus ~ 20 contributors
- 14. Why is Kafka so fast? • Fast writes: • While Kafka persists all data to disk, essentially all writes go to the page cache of OS, i.e. RAM. • Fast reads: • Very efficient to transfer data from page cache to a network socket • Linux: sendfile() system call • Combination of the two = fast Kafka! • Example (Operations):On a Kafka cluster where the consumers are mostly caught up you will see no read activity on the disks as they will be serving data entirely from cache. 14 1 http://kafka.apache.org/documentation.html#persistence
- 15. Flafka – Flume meets Kafka 15
- 16. Confluent - Centralized Ingestion with Kafka Pipeline 16
- 18. RealTime Stream Processing • Processing system • Apache Storm • Apache Samza • Apache Spark (Streaming) • Project Apex - DataTorrent • Storage • Hive HDFS • Hbase • MySql • Custom • Access • Depend of data storage • Scalable query interface - Kafka 18
- 19. Streaming Design Patterns • Micro batching • Unpredictable incoming data • Creating multiple streams • Out of sequence events • Stream joins • Top N metrics • External Lookup 19
- 20. ThankYou 20