Introduction of MapReduce
1 like780 views
The document describes MapReduce, a programming model for processing large datasets in a distributed environment. MapReduce allows users to write map and reduce functions, hiding the complexity of parallelization, fault tolerance, and load balancing. It works by dividing the input data into mapped key-value pairs, shuffling and sorting by key, and reducing the values for each key. This makes it easy to write distributed programs for tasks like inverted indexing, sorting, and counting URL frequencies. The implementation assigns tasks to worker nodes, handles failures, and optimizes for locality and load balancing.
Introduction of MapReduce
- 2. Introduction ● Motivation: lots of special-purpose programs should process large amounts of raw data – crawl(analyze) documents, web request logs, etc. – should use lots of machine to reduce processing time – implementation is time-consuming and complex ● Solution: Design a programming model– MapReduce – Hides the details of parallelization, fault-tolerance, locality optimization, and load balancing.
- 3. Map and Reduce ● Divide, Conquer, and Combine -> Divide, Maps, and Reduces ● User only need to implement Map() and Reduce() functions!(and some arguments)
- 4. Programming Model • Take an input pair • produces a set of intermediate key/value pairs Map • sorted intermediate pairs by key value • groups together all intermediate values with the same intermediate key Shuffling • Take intermediate key and value set of key • merges together these value Reduce
- 6. More Example ● Inverted Index – Find specified word in set of files – Input: <files(splited), docID> – Intermediate: <word, docID> – Final: <word, list<docID>> ● Distributed Grep ● Distributed Sort ● Count of URL Access Frequency ● Term-Vector per Host
- 7. More Example Distributed Grep Distributed Sort Inverted Index Intermediate: <word, docID> → Final: <word, list<docID>> Count of URL Access Frequency <URL, 1> → <URL, TotalCount> Term-Vector per Host summarize the most important words in docs <term, freq> → vector<term, freq>
- 9. Implementation ● Parallelization – Input of Map: partitioning the input data into M splits – Input of Reduce: partitioning intermediate data into R files ● Master program: assign M map tasks and R reduce tasks to worker programs – Map workers: Intermediate key/value pairs are written to local disk. Locations of these files would pass back to master. – Reduce workers: Get location from master and use remote procedure calls(RPC) to read data in local disk.
- 10. Fault Tolerance ● Worker Failure – ping every worker periodically – tasks in failed machine : ● rescheduling now assigning task ● reset completed map tasks and rescheduling ● but completed reduce tasks don't need to reset (files are stored in global file system) ● Master Failure – failure of master is unlikely – aborts the MapReduce, clients should check and retry it
- 11. Backup Tasks ● Straggler : machine that takes an unusually long time to complete tasks – bad disk – other tasks ● Solution: when MapReduce is close to completion, master schedules backup executions of the remaining in-progress tasks – Only wait one of they to complete – Takes 30% less time to complete, with computational resources increase by no more than a few percent
- 12. Performance ● Environment – 1800 machines – two 2GHz Intel Xeon processors with Hyper-Threading enabled, 4GB of memory, two 160GB IDE disks, and a gigabit Ethernet link. ● Sorts approximately 1TB of data – 891sec ● intentionally killed 200 out of 1746 workers several minutes – 933sec (just 5% increase) ● No backup tasks – 1283sec (44% increase)
- 13. 0 200 400 600 800 1000 1200 1400 Normal Fault Tolerance Without Backup task Performance Execute time(Sec)
- 14. Advantage ● Large variety of problems are easily expressible as MapReduce – Every work which can be divided! ● Easy to use for programmers who have no experience with distributed or parallel systems – What you think is how to deal with splited data, and how to compose result ● Code is simpler, easier to understand and modify
- 15. Application ● large-scale machine learning ● extraction of data used to produce reports of popular queries ● extraction of properties of web pages – PageRank ● Open Source implementation – Hadoop
- 16. Refinements ● Locality optimization –Input file copies in local disks ● Skipping Bad Records – ignore a few bad records, when doing statistical analysis on a large data set. – signal handler (When error, send information to master) ● Counter object – Piggybacked on the ping response
- 17. Refinements ● Locality optimization – Input file copies in local disks ● Intermediate key/value pairs is in order – Utilized sort and random access ● Input and Output Types – the key is the offset in the file and the value is the contents of the line. – reader interface ● Skipping Bad Records – ignore a few bad records, when doing statistical analysis on a large data set. – signal handler (When error, send information to master) ● Counter object – Piggybacked on the ping response ● Combiner function – partial merging at local disk before sending record
- 18. Reference Jeffrey Dean and Sanjay Ghemawat, 2004, MapReduce: Simplified Data Processing on Large Clusters