Big Data, Fast Data - MapReduce in Hazelcast
5 likes2,187 views
Christoph Engelbert will demonstrate how to use MapReduce in Hazelcast for distributed computing. MapReduce allows processing and transforming large datasets in a distributed fashion, and was popularized by Google. It works by mapping inputs to intermediate key-value pairs, reducing those values based on keys, and allows parallel processing across clusters. Hazelcast is an in-memory data grid that can be used as a distributed computing platform and implements MapReduce to allow distributed processing of large datasets across server clusters.
![Multiple values are combined to an
intermediate resultto preserve traffic
SOME PSEUDO CODE (2/3)
COMBINING
combine( word:String, counts:List[Int] ):Void ->
emit( word, sum( counts ) )
www.hazelcast.com](https://image.slidesharecdn.com/bigdatafastdata-usingmapreduceinhazelcast-140519124617-phpapp01/85/Big-Data-Fast-Data-MapReduce-in-Hazelcast-23-320.jpg)
![Values are reduced /aggregated to the requested result
SOME PSEUDO CODE (3/3)
REDUCING
reduce( word:String, counts:List[Int] ):Int ->
return sum( counts )
www.hazelcast.com](https://image.slidesharecdn.com/bigdatafastdata-usingmapreduceinhazelcast-140519124617-phpapp01/85/Big-Data-Fast-Data-MapReduce-in-Hazelcast-24-320.jpg)
![FOR MATHEMATICIANS
Process: (K x V)*→ (L x W)* ⇒ [(l1, w1), …, (lm, wm)]
Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)]
Reducing: L x W*→ X* ⇒ (l, [w1, …, wn]) → [x1, …,xn]
www.hazelcast.com](https://image.slidesharecdn.com/bigdatafastdata-usingmapreduceinhazelcast-140519124617-phpapp01/85/Big-Data-Fast-Data-MapReduce-in-Hazelcast-25-320.jpg)
Big Data, Fast Data - MapReduce in Hazelcast
- 1. BIG DATA - FAST DATA USING MAPREDUCE IN HAZELCAST Source:http://www.newscientist.com/gallery/dn17805-computer-museums-of-the-world/11 www.hazelcast.com
- 2. WHO AM I Christoph Engelbert(@noctarius2k) 8+ years of JavaWeirdoness Performance, GC, traffic topics Apache DirectMemoryPMC Previous companies incl. Ubisoftand HRS CastMapRMapReduce for Hazelcast3 www.hazelcast.com
- 4. HAZELCAST A SHORT SPACE TRIP www.hazelcast.com
- 5. WHAT IS HAZELCAST? In-MemoryData-Grid DataPartioning(Sharding) JavaCollections Implementation Distributed ComputingPlatform www.hazelcast.com
- 8. TREND OF PRICES DataSource:http://www.jcmit.com/memoryprice.htm www.hazelcast.com
- 10. DISTRIBUTED COMPUTING OR MULTICORE CPU ON STEROIDS www.hazelcast.com
- 11. THE IDEA OF DISTRIBUTED COMPUTING Source:https://www.flickr.com/photos/stefan_ledwina/1853508040 www.hazelcast.com
- 13. MULTICORE IS NOT NEW Source:http://en.wikipedia.org/wiki/File:80386with387.JPG www.hazelcast.com
- 17. MAP & REDUCE THE BLACK MAGIC FROM PLANET GOOGLE www.hazelcast.com
- 18. USE CASES LogAnalysis DataQuerying Aggregation and summing Distributed Sort ETL (ExtractTransform Load) and more... www.hazelcast.com
- 22. Dataare mapped /transformed in asetof key-value pairs SOME PSEUDO CODE (1/3) MAPPING map( key:String, document:String ):Void -> for each w:word in document: emit( w, 1 ) www.hazelcast.com
- 23. Multiple values are combined to an intermediate resultto preserve traffic SOME PSEUDO CODE (2/3) COMBINING combine( word:String, counts:List[Int] ):Void -> emit( word, sum( counts ) ) www.hazelcast.com
- 24. Values are reduced /aggregated to the requested result SOME PSEUDO CODE (3/3) REDUCING reduce( word:String, counts:List[Int] ):Int -> return sum( counts ) www.hazelcast.com
- 25. FOR MATHEMATICIANS Process: (K x V)*→ (L x W)* ⇒ [(l1, w1), …, (lm, wm)] Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)] Reducing: L x W*→ X* ⇒ (l, [w1, …, wn]) → [x1, …,xn] www.hazelcast.com
- 26. MAPREDUCE PROGRAMS IN GOOGLE SOURCE TREE Source:http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html www.hazelcast.com