Flink Streaming Hadoop Summit San Jose
Download as PPTX, PDF8 likes2,383 views
This document provides an overview of Apache Flink, an open-source stream processing framework. It discusses the rise of stream processing and how Flink enables low-latency applications through features like pipelining, operator state, fault tolerance using distributed snapshots, and integration with batch processing. The document also outlines Flink's roadmap, which includes graduating its DataStream API, fully managing windowing and state, and unifying batch and stream processing.
![DataStream API
25
case class Word (word: String, frequency: Int)
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS))
.groupBy("word").sum("frequency")
.print()
val lines: DataSet[String] = env.readTextFile(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.groupBy("word").sum("frequency")
.print()
DataSet API (batch):
DataStream API (streaming):](https://image.slidesharecdn.com/flinkstreaminghadoopsummitsanjoseweb-150612010602-lva1-app6892/85/Flink-Streaming-Hadoop-Summit-San-Jose-25-320.jpg)
Flink Streaming Hadoop Summit San Jose
- 1. Stream processing with Apache Flink™ Kostas Tzoumas @kostas_tzoumas
- 2. The rise of stream processing 2
- 3. Why streaming 3 Data Warehouse Batch Data availability Streaming - Strict schema - Load rate - BI access - Some schema - Load rate - Programmable - Some schema - Ingestion rate - Programmable 2008 20152000 - Which data? - When? - Who?
- 4. What does streaming enable? 1. Data integration 2. Low latency applications 4 • Fresh recommendations, fraud detection, etc • Internet of Things, intelligent manufacturing • Results “right here, right now” cf. Kleppmann: "Turning the DB inside out with Samza" 3. Batch < Streaming
- 5. New stack next to/inside Hadoop 5 Files Batch processors High-latency apps Event streams Stream processors Low-latency apps
- 7. Stream platform architecture 7 - Gather and backup streams - Offer streams for consumption - Provide stream recovery - Analyze and correlate streams - Create derived streams and state - Provide these to upstream systems Server logs Trxn logs Sensor logs Upstream systems
- 10. What is Flink 10 Gelly Table ML SAMOA DataSet (Java/Scala) DataStream (Java/Scala) HadoopM/R Local Cluster Yarn Tez Embedded Dataflow Dataflow(WiP) MRQL Table Cascading(WiP) Streaming dataflow runtime Storm(WiP) Zeppelin
- 11. Motivation for Flink 11 An engine that can natively support all these workloads. Flink Stream processing Batch processing Machine Learning at scale Graph Analysis
- 12. Stream processing in Flink 12
- 13. What is a stream processor? 1. Pipelining 2. Stream replay 3. Operator state 4. Backup and restore 5. High-level APIs 6. Integration with batch 7. High availability 8. Scale-in and scale-out 13 Basics State App development Large deployments See http://data-artisans.com/stream-processing-with-flink.html
- 14. Pipelining 14 Basic building block to “keep the data moving” Note: pipelined systems do not usually transfer individual tuples, but buffers that batch several tuples!
- 15. Operator state User-defined state • Flink transformations (map/reduce/etc) are long-running operators, feel free to keep around objects • Hooks to include in system's checkpoint Windowed streams • Time, count, data-driven windows • Managed by the system (currently WiP) Managed state (WiP) • State interface for operators • Backed up and restored by the system with pluggable state backend (HDFS, Ignite, Cassandra, …) 15
- 16. Streaming fault tolerance Ensure that operators see all events • “At least once” • Solved by replaying a stream from a checkpoint, e.g., from a past Kafka offset Ensure that operators do not perform duplicate updates to their state • “Exactly once” • Several solutions 16
- 17. Exactly once approaches Discretized streams (Spark Streaming) • Treat streaming as a series of small atomic computations • “Fast track” to fault tolerance, but does not separate business logic from recovery MillWheel (Google Cloud Dataflow) • State update and derived events committed as atomic transaction to a high-throughput transactional store • Needs a very high-throughput transactional store Chandy-Lamport distributed snapshots (Flink) 17
- 18. Distributed snapshots in Flink Super-impose checkpointing mechanism on execution instead of using execution as the checkpointing mechanism 18
- 19. 19 JobManager Register checkpoint barrier on master Replay will start from here
- 20. 20 JobManagerBarriers “push” prior events (assumes in-order delivery in individual channels) Operator checkpointing starting Operator checkpointing finished Operator checkpointing in progress
- 21. 21 JobManager Operator checkpointing takes snapshot of state after data prior to barrier have updated the state. Checkpoints currently one-off and synchronous, WiP for incremental and asynchronous State backup Pluggable mechanism. Currently either JobManager (for small state) or file system (HDFS/Tachyon). WiP for in-memory grids
- 22. 22 JobManager Operators with many inputs need to wait for all barriers to pass before they checkpoint their state
- 23. 23 JobManager State snapshots at sinks signal successful end of this checkpoint At failure, recover last checkpointed state and restart sources from last barrier guarantees at least once State backup
- 24. Benefits of Flink’s approach Data processing does not block • Can checkpoint at any interval you like to balance overhead/recovery time Separates business logic from recovery • Checkpointing interval is a config parameter, not a variable in the program (as in discretization) Can support richer windows • Session windows, event time, etc Best of all worlds: true streaming latency, exactly-once semantics, and low overhead for recovery 24
- 25. DataStream API 25 case class Word (word: String, frequency: Int) val lines: DataStream[String] = env.fromSocketStream(...) lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS)) .groupBy("word").sum("frequency") .print() val lines: DataSet[String] = env.readTextFile(...) lines.flatMap {line => line.split(" ") .map(word => Word(word,1))} .groupBy("word").sum("frequency") .print() DataSet API (batch): DataStream API (streaming):
- 26. Roadmap Short-term (3-6 months) • Graduate DataStream API from beta • Fully managed window and user-defined state with pluggable backends • Table API for streams (towards StreamSQL) Long-term (6+ months) • Highly available master • Dynamic scale in/out • FlinkML and Gelly for streams • Full batch + stream unification 26
- 27. Closing 27
- 28. tl;dr: what was this about? Streaming is the next logical step in data infrastructure Many new "fast data" platforms are being built next to or inside Hadoop – will need a stream processor The case for Flink as a stream processor • Proper engine foundation • Attractive APIs and libraries • Integration with batch • Large (and growing!) community 28
- 29. Apache Flink: community 29 One of the most active big data projects after one year in the Apache Software Foundation
- 30. I Flink, do you? 30 If you find this exciting, get involved and start a discussion on Flink‘s mailing list, or stay tuned by subscribing to news@flink.apache.org, following flink.apache.org/blog, and @ApacheFlink on Twitter
- 31. 31 flink-forward.org Spark & Friends meetup June 16 Bay Area Flink meetup June 17
Editor's Notes
- #8: What are the technologies that enable streaming? The open source leaders in this space is Apache Kafka (that solves the integration problem), and Apache Flink (that solves the analytics problem, removing the final barrier). Kafka and Flink combined can remove the batch barriers from the infrastructure, creating a truly real-time analytics platform.
- #30: Other data points Google (cloud dataflow) Hortonworks Cloudera Adatao Concurrent Confluent We have been part of this open source movement with Apache Flink. Flink is a streaming dataflow engine that can run in Hadoop clusters. Flink has grown a lot over the past year both in terms of code and community. We have added domain-specific libraries, a streaming API with streaming backend support, etc, etc. Tremendous growth. Flink has also grown in community. The project is by now a very established Apache project, it has more than 140 contributors (placing it at the top 5 of Apache big data projects), and several companies are starting to experiment with it. At data Artisans we are supporting two production installations (ResearchGate and Bouygues Telecom), and are helping a number of companies that are testing Flink (e.g., Spotify, King.com, Amadeus, and a group at Yahoo). Huawei and Intel have started contributing to Flink, and interest in vendors is picking up (e.g., Adatao, Huawei, Hadoop vendors). All of this is the result of purely organic growth with very little marketing investment from data Artisans.