Map reduce and Hadoop on windows
Download as PPTX, PDF2 likes2,592 views
Map Reduce is a parallel and distributed approach developed by Google for processing large data sets. It has two key components - the Map function which processes input data into key-value pairs, and the Reduce function which aggregates the intermediate output of the Map into a final result. Input data is split across multiple machines which apply the Map function in parallel, and the Reduce function is applied to aggregate the outputs.
Map reduce and Hadoop on windows
- 1. Map Reduce Muhammad UsmanShahidSoftware Engineer Usman.shahid.st@hotmail.com10/17/20111
- 2. Parallel ProgrammingUsed for performance and efficiency.Processing is broken up into parts and done concurrently.Instruction of each part run on a separate CPU while many processors are connected.Identification of set of tasks which can run concurrently is important.A Fibonacci function is Fk+2 = Fk + Fk+1.It is clear that Fibonacci function can not be parallelized as each computed value depends on previous.Now consider a huge array which can be broken up into sub-arrays.10/17/20112
- 3. Parallel Programming10/17/20113If each element required some processing, with no dependencies in the computation, we have an ideal parallel computing opportunity.
- 4. Google Data CenterGoogle believes buy cheap computers but numerous in number.Google has parallel processing concept in its data centers.Map Reduce is a parallel and distributed approach developed by Google for processing large data sets. 10/17/20114
- 5. Map Reduce IntroductionMap Reduce has two key components. Map and Reduce.Map function is used on input values to calculate a set of key/Value pairs.Reduce aggregates this data into a scalar.10/17/20115
- 6. Data Distribution Input files are split into M pieces on distributed file systems.Intermediate files are created from map tasks are written to local disks.Output files are written to distributed file systems.10/17/20116
- 8. Map Reduce FunctionMap Reduce function by an example see the query “Select Sum(stuMarks) from student group by studentSection”.In above query “select” phase is doing the same as Map do and “Group By” same as Reduce Phase.10/17/20118
- 9. Classical ExampleThe classical example of Map Reduce is the log file analysis.Big log files are split and mapper search for different web pages which are accessed.Every time a web page is found in the log a key/value pair is emitted to the reducer in such way that key = web page and value = 1.The reducer aggregates the number for a certain web pages. Result is the count of total hits for each web page.10/17/20119
- 10. Reverse Web Link GraphIn this example Map function outputs (URL target, source) from an input web page (source).Reduce function concatenates the list of all source URL(s) with a give target of URL and returns (target, list(source)).10/17/201110
- 11. Other Examples Map Reduce can be used for the lot of problems.For Example the Google used the Map Reduce for the calculation of page ranks.Word count in large set of documents can also be resolved by Map Reduce very efficiently.Google library for the Map Reduce is not open source but an implementation in java called hadoop is an open source.10/17/201111
- 12. Implementation of ExampleWord Count is a simple application that counts the number of occurrences of words in a given set of inputs.Hadoop library is used for its implementation.Code is given in the below attached file.10/17/201112
- 13. Usage of ImplementationFor example the input files are $ bin/hadoopdfs -ls /usr/joe/wordcount/input/ /usr/joe/wordcount/input/file01 /usr/joe/wordcount/input/file02 $ bin/hadoopdfs -cat /usr/joe/wordcount/input/file01 Hello World Bye World $ bin/hadoopdfs -cat /usr/joe/wordcount/input/file02 Hello Hadoop Goodbye HadoopRun the application.Word Count is straight forward problem.10/17/201113
- 14. Walk Through ImplementationThe Mapper implementation (lines 14-26), via the map method (lines 18-25), processes one line at a time, as provided by the specified TextInputFormat (line 49). It then splits the line into tokens separated by whitespaces, via the StringTokenizer, and emits a key-value pair of < <word>, 1>.For the given sample input the first map emits:< Hello, 1> < World, 1> < Bye, 1> < World, 1> The second map emits:< Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> 10/17/201114
- 15. Walk Through ImplementationWordCount also specifies a combiner (line 46). Hence, the output of each map is passed through the local combiner (which is same as the Reducer as per the job configuration) for local aggregation, after being sorted on the keys.The output of the first map:< Bye, 1> < Hello, 1> < World, 2> The output of the second map:< Goodbye, 1> < Hadoop, 2> < Hello, 1> 10/17/201115
- 16. Walk Through ImplementationThe Reducer implementation (lines 28-36), via the reduce method (lines 29-35) just sums up the values, which are the occurence counts for each key (i.e. words in this example).Thus the output of the job is:< Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> The run method specifies various facets of the job, such as the input/output paths (passed via the command line), key/value types, input/output formats etc., in theJobConf. It then calls the JobClient.runJob (line 55) to submit the and monitor its progress.10/17/201116
- 18. Map Reduce ExecutionMap Reduce library is the user program that first splits the input files in M pieces. Then it start ups many copies of the program on cluster of machines.One of the copy is special – The Master other are the workers. There are M Map tasks and R Reduce tasks to assign. The master picks the idle workers and assign them the Map task or Reduce Task.A worker who is assigned Map task reads the contents of corresponding input split. It parses the key value pair and pass it to user defined Map function this generates the intermediate key/value pairs buffered in the memory.Periodically, the buffered pairs are written to local disks. The locations of these buffered pairs on local disks are passed back to the master, who is responsible for forwarding them to the reducer workers.10/17/201118
- 19. Map Reduce Execution When master notify a reduce worker about these location, it uses RPC to access this local data, then it sorts the data.The reduce worker iterates over the sorted intermediate data, for each unique key it passes the key and values to the reduce function. The output is appended to the final output file.Many associated issues are handled by the library likeParallelizationFault Tolerance Data DistributionLoad Balancing10/17/201119
- 20. DebuggingOffer human readable status info on http server, user can see jobs In progress, Completed etc.Allows use of GDB and other debugging tools.10/17/201120
- 21. ConclusionsSimplifies large scale computations that fit this model.Allows user to focus on the problem without worrying about the details.It is being used by renowned companies like Google and Yahoo.Google library for Map Reduce is not open source but a project of Apache called hadoop is an open source library for Map Reduce.10/17/201121