Flink September 2015 Community Update
Download as PPTX, PDF2 likes833 views
This document summarizes the September 2015 community update for Apache Flink. Key highlights include Matthias Sax joining as a new committer, the release of version 0.9.1, and discussions starting around releasing version 0.10. Version 0.10 will include improvements to window operators, memory allocation, and new connectors to HDFS, Elasticsearch, and Kafka. The community held various meetups and presentations around the world in September and Flink was recognized as one of the best open source big data tools.
![Articles
• data Artisans blog: Kafka + Flink: A practical, how-to guide
[1]
• Gartner blog: Apache Flink Offers a Challenge to Spark [2]
• data Artisans blog: Batch is a special case of streaming [3]
• Flink blog: Off-heap Memory in Apache Flink and the
curious JIT compiler [4]
• MapR blog: Apache Flink: A New Way to Handle Streaming
Data [5]
• Big Data Knowledge Base: Happenings in the Flink
Community - September 2015 [6]
6
[1] http://data-artisans.com/kafka-flink-a-practical-how-to/
[2] http://blogs.gartner.com/nick-heudecker/apache-flink-offers-a-challenge-to-spark/
[3] http://data-artisans.com/batch-is-a-special-case-of-streaming/
[4] http://flink.apache.org/news/2015/09/16/off-heap-memory.html
[5] https://www.mapr.com/blog/apache-flink-new-way-handle-streaming-data
[6] http://sparkbigdata.com/102-spark-blog-slim-baltagi/17-happenings-in-the-flink-community-september-2015](https://image.slidesharecdn.com/flinkcommunityupdateslideshare-151004222519-lva1-app6892/85/Flink-September-2015-Community-Update-7-320.jpg)
Flink September 2015 Community Update
- 1. Berlin Apache Flink Meetup #11 Community Update September 2015 Robert Metzger Committer and PMC Member rmetzger@apache.org @rmetzger_
- 2. Apache Flink is an open source platform for scalable batch and stream data processing. Apache Flink is … flink.apache.org 1 • The core of Flink is a distributed streaming dataflow engine. • Executing dataflows in parallel on clusters • Providing a reliable foundation for various workloads • DataSet and DataStream programming abstractions are the foundation for user programs and higher layers
- 3. One engine for many use cases flink.apache.org 2 Real time streaming topologies Machine Learning at scale Graph Analysis Long batch pipelines
- 4. What happened? • New Committer: Matthias Sax • 0.9.1 released • Discussions for releasing 0.10 started • Cascading on Flink released: https://github.com/dataArtisans/cascading-flink • Flink+NiFi integration pull request opened flink.apache.org 3
- 5. Now in master (0.10-SNAPSHOT) flink.apache.org 4 • Flink dropped Hadoop 2.2.0 support (we require 2.3.0) • Scala 2.11 artifacts are now available • Support for allocating off-heap memory • New window operators (general purpose and processing time windows) • old implementation: 50K / core / sec (gets slower over time, high GC overhead) • new implementation w/o pre-aggregation: 800K / sec / core (moderate GC overhead) • new implementation w/ pre-aggregation: 3mio / sec / core (low GC overhead) • Rolling HDFS file sink for DataStream API • Sink for ElasticSearch • New JobManager dashboard • New FlinkKafkaProducer
- 6. Flink among “The best open source big data tools” flink.apache.org 5
- 7. Articles • data Artisans blog: Kafka + Flink: A practical, how-to guide [1] • Gartner blog: Apache Flink Offers a Challenge to Spark [2] • data Artisans blog: Batch is a special case of streaming [3] • Flink blog: Off-heap Memory in Apache Flink and the curious JIT compiler [4] • MapR blog: Apache Flink: A New Way to Handle Streaming Data [5] • Big Data Knowledge Base: Happenings in the Flink Community - September 2015 [6] 6 [1] http://data-artisans.com/kafka-flink-a-practical-how-to/ [2] http://blogs.gartner.com/nick-heudecker/apache-flink-offers-a-challenge-to-spark/ [3] http://data-artisans.com/batch-is-a-special-case-of-streaming/ [4] http://flink.apache.org/news/2015/09/16/off-heap-memory.html [5] https://www.mapr.com/blog/apache-flink-new-way-handle-streaming-data [6] http://sparkbigdata.com/102-spark-blog-slim-baltagi/17-happenings-in-the-flink-community-september-2015
- 8. Events in September flink.apache.org 7 VLDB 2015 Conference Workshop Flink Training in Berlin Washington DC Meetup Meetup in Belgium Milwaukee Meetup Budapest: 2 ApacheCon Talks BigTop Workshop data2day Conference in Karlsruhe Chicago Meetup
- 11. flink.apache.org 10 Flink Forward: 2 days conference with free training in Berlin, Germany • Schedule: http://flink-forward.org/?post_type=day
Editor's Notes
- #11: in 3 month