"MapReduce: Simplified Data Processing on Large Clusters" Paper Presentation @ Papers We Love Bucharest
Download as PPTX, PDF2 likes591 views
MapReduce is a programming model and implementation for processing large datasets across clusters of computers. It allows users to specify map and reduce functions. The map function processes input key-value pairs to generate intermediate pairs, while the reduce function combines intermediate values into final output. Google developed MapReduce to simplify distributed computing on large datasets, addressing issues like parallelization, fault tolerance, and load balancing. It works by splitting input data into blocks and assigning them to worker nodes that apply the user-defined map and reduce functions to process the data in parallel.
![What is?
âMapReduce is a programming model and implementation for processing large data sets
âUsers specify a Map function that processes a key-value pair to generate a set of
intermediate key-value pairs and a Reduce function that aggregates all intermediate
values that share the same intermediate key in order to combine the derived data
appropriately
map:(k1, v1)->[(k2, v2)]
reduce:(k2, [v2])->[(k3, v3)]](https://image.slidesharecdn.com/pwlb1510-mapreduce-151013124417-lva1-app6892/85/MapReduce-Simplified-Data-Processing-on-Large-Clusters-Paper-Presentation-Papers-We-Love-Bucharest-3-320.jpg)
![MapReduce model in practice
map(String key, String value):
for each Word w in value:
EmitIntermediate(w, "1");
reduce(String key, Iterator values)
int result = 0;
for each v in values:
result += ParseInt(v);
Emit(key, AsString(result));
map:(k1, v1)->[(k2, v2)]
reduce:(k2, [v2])->[(k3, v3)]](https://image.slidesharecdn.com/pwlb1510-mapreduce-151013124417-lva1-app6892/85/MapReduce-Simplified-Data-Processing-on-Large-Clusters-Paper-Presentation-Papers-We-Love-Bucharest-13-320.jpg)
"MapReduce: Simplified Data Processing on Large Clusters" Paper Presentation @ Papers We Love Bucharest
- 1. MapReduce: Simplified Data Processing on Large Clusters Papers We Love Bucharest Chapter October 12, 2015
- 2. I am Adrian Florea Architect Lead @ IBM Bucharest Software Lab Hello!
- 3. What is? âMapReduce is a programming model and implementation for processing large data sets âUsers specify a Map function that processes a key-value pair to generate a set of intermediate key-value pairs and a Reduce function that aggregates all intermediate values that share the same intermediate key in order to combine the derived data appropriately map:(k1, v1)->[(k2, v2)] reduce:(k2, [v2])->[(k3, v3)]
- 4. Map âfunction (mathematics) âmap (Java) âSelect (.NET) Where else have we seen this? Reduce âfold (functional programming) âreduce (Java) âAggregate (.NET)
- 5. Other analogies "To draw an analogy to SQL, map is like the group-by clause of an aggregate query. Reduce is analogous to the aggregate function that is computed over all the rows with the same group- by attribute" D.J. DeWitt & M. Stonebraker Divide-and-conquer algorithms ârecursively breaking down a problem into two or more sub-problems of the same (or related) type (divide), until these become simple enough to be solved directly (conquer). The solutions to the sub-problems are then combined to give a solution to the original problem.â
- 6. Jeffrey Dean Google Fellow December 6, 2004 4PM Sanjay Ghemawat Google Fellow
- 7. History âApril 1960: John McCarthy introduced the concept of âmaplistâ âSeptember 4, 1998: Google founded â1998-2003: hundreds of special-purpose large data computation programs in Google âFebruary 2003: 1st version of MapReduce âAugust 2003: MapReduce significant enhancements âJune 18, 2004: Patent US7650331 B1 filed âDecember 6, 2004: 1st MapReduce public presentation â2005: Hadoop implementation started in Java (Douglass R. Cutting & Michael J. Cafarella) âSeptember 4, 2007: Hadoop 0.14.1 âJanuary 19, 2010: Patent US7650331 B1 published âJuly 6, 2015: Hadoop 2.7.1
- 8. Distribution issues âCommunication and routing which nodes should be involved? what transport protocol should be used? threads/events/connections management remote execution of your processing code? âFault tolerance and fault detection âLoad balancing / partitioning of data heterogeneity of nodes skew in data network topology âParallelization strategy algorithmic issues of work splitting âwithout having to deal with failures, the rest of the support code just isnât so complicateâ S. Ghemawat
- 9. MapReduce model Map Operation Reduce Operation Input Data Intermediate Data Output Data
- 10. application-independent Map Module application-independent Reduce Module MapReduce system application-specific Map Operation application-specific Reduce OperationInput Data Intermediate Data Output Data
- 11. Original article Hadoop wiki
- 12. Original article Hadoop wiki
- 13. MapReduce model in practice map(String key, String value): for each Word w in value: EmitIntermediate(w, "1"); reduce(String key, Iterator values) int result = 0; for each v in values: result += ParseInt(v); Emit(key, AsString(result)); map:(k1, v1)->[(k2, v2)] reduce:(k2, [v2])->[(k3, v3)]
- 14. How long does it take to go through 1 TB? Sequentially: 3 hours MapReduce: startup overhead 70 seconds + computation 80 seconds Environment â 1800 Linux dual-processorx86 machines, 2-4 GB memory â Fast Ethernet/Giga Ethernet â Inexpensive IDE disks and a distributed Google File System
- 15. Take a walk through a Google data center
- 16. Tianhe-2, #1 supercomputer: 3,120,000 cores, 1,5 PB total memory
- 17. Execution diagram âMaster process is a task itself initiated by the WQM and is responsible for assigning all other tasks to worker processes âEach worker invokes at least a map thread and a reduce thread âIf a worker fails, its tasks are reassigned to another worker process âWhen WQM receives a job, it allocates the job to the master that calculates and requires M+R+1 processes to be allocated to the job âWQM responds with the process allocation info (can result less processes) to the master that will manage the performance of the job âReduce tasks begin work when the master informs them that there are intermediate files ready âInput data (files/DB/memory) are splitted in data blocks (16-64 MB) automatically or configurable âThe worker to which a map task has been assigned applies the map() operator to the respective input data block âWhen the worker completes the task, it informs the master of the status âMaster informs workers where to find intermediate data and schedules their reading âWorkers (3 & 4) sort the intermediate key-value pairs, then merge (by applying reduce()) them and write to output
- 18. Workflow diagram âWhen a process completes a task it informs WQM which updates the status tables âWhen WQM discovers one process failed, it assign its tasks to a new process and updates the status tables Task Status Table âTaskID âStatus (InProgress, Waiting, Completed, Failed) âProcessID âInputFiles (Input, Intermediate) âOutputFiles Process Status Table âProcessID âStatus (Idle, Busy, Failed) âLocation (CPU ID, etc.) âCurrent (TaskID, WQM)
- 19. Questions from the audience @ original paper presentation âQ: Wanted to know of any task that could not be handled using MapReduce? A: join operations could not be performed with the current model âQ: Wondered how MapReduce differs from parallel databases? A: MapReduce is stored across a large number of machines as compared to parallel databases, the abstractions are fairly simple to use in MapReduce, and MapReduce also benefits greatly from locality optimizations
- 20. Bibliography â J. Dean, S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", OSDI'04, Dec. 6, 2004 â J. Dean, S. Ghemawat, âSystem and method for efficient large-scale data processingâ, Patent US 7650331 B1, Jan. 19, 2010 â S. Ghemawat, J. Dean, J. Zhao, M. Austern, A. Spector, "Google Technology RoundTable: Map Reduce", Aug. 21, 2008 â Youtube â P. Mahadevan, "OSDI'04 Conference Reports", ;LOGIN: Vol. 30, No. 2, Apr. 2005, p. 61 â R. Jacotin, âLecture: The Google MapReduceâ, SlideShare, October 3, 2014
- 21. Any questions ? You can find me at Thanks!