[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017
Download as PPTX, PDF0 likes353 views
This slide deck discusses distributed data processing using Java 8 Streams and Hazelcast Jet. It provides an overview of Java Streams and their limitations for large datasets. It then introduces Hazelcast IMDG for distributed caching and collections. Hazelcast Jet is presented as a distributed data processing framework built on Hazelcast IMDG that can be used to distribute and parallelize stream processing. Examples of distributed streams and the directed acyclic graph (DAG) model used by Jet are shown.
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-2-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-11-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-12-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-13-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-17-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-22-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-23-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-24-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-26-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-31-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-33-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-35-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-38-320.jpg)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://image.slidesharecdn.com/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616/85/NYJavaSig-Riding-the-Distributed-Streams-Feb-2nd-2017-42-320.jpg)
[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017
- 1. This slide blank on purpose
- 3. > whoami • Solutions Architect @Hazelcast • Hang out with awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
- 4. Agenda • Refreshing knowledge on Java 8 Streams • Distribute and Conquer • Distributed Data • Distributed Streams • How we did all this
- 6. Java 8 Streams… • An abstraction represents a sequence of elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
- 7. Why I should care about Stream API? • You’re Java developer
- 8. What does regular Java developer think about Scala? advanced
- 9. Why I should care about Stream API? • You’re Java developer • Many Java developers know Java • It’s all about data processing
- 10. java.util.stream operations • map(), flatMap(), filter() • reduce(), collect() • sorted()
- 14. Problem • One does not simply put all Big Data in one machine
- 15. Problem • Data doesn’t fit just one machine
- 16. Problem • One does not simply put all Big Data in one machine • Data is too important to have it only one machine
- 18. CACHES
- 20. Solution • Use Distributed Map aka IMap
- 21. What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2 Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
- 28. 28
- 29. Solution • serializable version of the interfaces • Introducing DistributedStream 29
- 30. 30
- 32. 32 Jet Streams
- 34. What’s Hazelcast Jet? • General purpose distributed data processing framework • Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 34
- 36. DAG 36
- 37. Job Execution 37
- 39. Future (It’s bright!) • Memory module for processing big data • Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
- 40. Your fuel, our Jet Engine • Public release – Feb 7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note viktor@hazelcast.com • Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
- 41. Conclusion • Java Stream API provides very white range of data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters
Editor's Notes
- #9: Credits to Jeka Borisov
- #13: http://www.nogarlicnoonions.com/images/article_images/2015-12/five-guys-dubai232015-12-28-11-24-23.jpg
- #19: Подумать на тему нарезанного барана
- #27: http://www.velvet.by/files/userfiles/19083/ekrk.jpg
- #32: http://www.velvet.by/files/userfiles/19083/ekrk.jpg
- #42: Интересный факт из войны и мира Что андрей и пьер действительно главные герои