Map Reduce introduction (google white papers)
- 2. A simple programming model Functional model For large-scale data processing Exploits large set of commodity computers Executes process in distributed manner Offers high availability
- 3. Lots of demands for very large scale data processing A certain common themes for these demands Lots of machines needed (scaling) Two basic operations on the input ▪ Map ▪ Reduce
- 4. Map: Accepts input key/value pair Emits intermediate key/value pair Reduce : Accepts intermediate key/value* pair Emits output key/value pair Very big data Result M A P R E D U C E Partitioning Function
- 5. Very big data Split data Split data Split data Split data grep grep grep grep matches matches matches matches cat All matches
- 8. Map Process a key/value pair to generate intermediate key/value pairs Reduce Merge all intermediate values associated with the same key Partition By default : hash(key) mod R Well balanced
- 9. No reduce can begin until map is complete Master must communicate locations of intermediate files Tasks scheduled based on location of data If map worker fails any time before reduce finishes, task must be completely rerun MapReduce library does most of the hard work for us!
- 10. User to do list: indicate: ▪ Input/output files ▪ M: number of map tasks ▪ R: number of reduce tasks ▪ W: number of machines Write map and reduce functions Submit the job
- 11. String Match, such as Grep Reverse index Count URL access frequency Lots of examples in data mining
- 13. Provide a general-purpose model to simplify large-scale computation Allow users to focus on the problem without worrying about details
- 14. Original paper (http://labs.google.com/papers/mapreduce.h tml) On wikipedia (http://en.wikipedia.org/wiki/MapReduce) Hadoop – MapReduce in Java (http://lucene.apache.org/hadoop/) http://code.google.com/edu/parallel/mapred uce-tutorial.html