サンプルから見るMap reduceコード
2 likes1,387 views
This document provides an overview of MapReduce and its components in Hadoop. It discusses key classes like Mapper, Reducer, MapTask, ReduceTask, and JobConf that are used to define and run MapReduce jobs. It also describes how mappers process input splits in parallel and generate output which is shuffled and sorted for input to the reducers.
サンプルから見るMap reduceコード
- 2. Cloudera Avro Sqoop Desktop Pig Hive HBase Chukwa Map Zoo HDFS Reduce Keeper Core
- 3. Cloudera Avro Sqoop Desktop Pig Hive HBase Chukwa Map Zoo HDFS Reduce Keeper Core
- 4. • MapReduce – Mapper/Reducer •
- 5. MapReduce • WordCount • • – Mapper/Reducer Job ⾏行行 – InputFormat/OutputFormat ⽅方 – HDFS(FileSystem) – Writable ⽅方
- 6. WordCount • Hadoop Hello World • API (org.apache.hadoop.mapreduce) • API
- 7. Grep • grep – grepJob/sortJob 2 ⾏行行 – JobConf/Mapper/Reducer ⽅方 – Mapper RegexMapper ⾏行行 <Text, Long> SequenceFileFormat – sortJob – ⼒力力 –
- 8. Grep - • JobConf • Mapper • Reducer
- 9. o.a.hadoop.mapred.JobConf • – mapred-default.xml – conf/mapred-site.xml – XML ⾝身 DOM – ⾃自 ⽬目 ⼿手 – ⼦子 • JobConf child = new JobConf( Conf, jar );
- 10. mapred-site.xml <configuration> <!– --> <property> <key>mapred.job.tracker</key> <value>your-site:9001</value> </property> </configuration>
- 11. o.a.hadoop.mapred.Mapper • Mapper • InputSplit Mapper • MapTask/MapRunner • map(KEY, VALUE, COLLECTOR, REPORTER) – KEY:Map VALUE:Map – COLLECTOR: – REPORTER: API • MapReduceBase
- 12. o.a.hadoop.mapred.MapTask • Map • initiazlize (Task Reducer ) – ⽣生 – (o.a.h.mapred.TaskStatus.State) • RUNNING, SUCCEEDED, FAILED, UNASSIGNED, KILLED, COMMIT_PENDING, FAILED_UNCLEAN, KILLED_UNCLEAN – OutputCommiter ⽣生 • Task ⼒力力 ⾏行行 • ⼒力力 – mapred.work.output.dir
- 13. o.a.h.mapred.MapTask cont • run runOldMapper • JobClient InputSplit • RecordReader
- 14. o.a.h.mapred.MapTask cont2 • Reduce – spill (* ) • $mapred.local.dir/taskTracker/jobcache/$ {taskid}/output/spill${spillNumber}.out – Reducer ⼒力力 • Combiner min.num.spills.for.combine combiner – RecordWriter ⼒力力 • MapRunner
- 15. o.a.h.mapred.MapRunner • MapRunnable – mapred.map.runner.class – Hadoop PipeMapRunner – Map MultiThreadedMapRunner
- 16. o.a.h.mapred.MapRunner cont • run(RecordReader, OutputCollector, Reporter) – RecordReader: InputFormat Split Reader(InputFormat/RecordReader ) • – RecordReader – ⾝身 –
- 17. MapTask MapRunner Mapper Record Output Reader Collector Input Split⽣生 Spill & run createKey() SpillThread createValue() next(key, value) EOF Map(key, value, Spill outputCollector, reporter)
- 18. m(_ _)m
- 19. • Mapper – JobConf – Mapper/MapRunner/MapTask • – Reducer • Reducer ⾏行行 • Reducer ⾏行行 – InputFormat/RecordReader
- 20. o.a.h.mapred.Reducer • Reducer • InputSplit Mapper • ReduceTask/ReduceRunner • reduce(KEY, Iterator<VALUE>, COLLECTOR, REPORTER) – KEY: Iterator<VALUE>: – COLLECTOR: – REPORTER: API • MapReduceBase
- 21. o.a.h.mapred.ReduceTask • SHUFFLE • ReduceTask.ReduceCopier – fetchOutputs( Merger.MergeQueue) • Map x mapred.reduce.parallel.copies – MapOutputCopier • Map ⾏行行 LocalFSMerger • ⾏行行 InMemFSMergeThread • GetMapEventsThread – Map – < , MapOutputLocation(taskId, host, httpUrl)> • ⼀一 TaskTracker ⼯工
- 22. o.a.h.mapred.ReduceTask • run(RecordReader, OutputCollector, Reporter) • SORT – Memory, disk ⽣生 • RowKeyValueItetator – Reducer ⽣生 – RecordWriter ⽣生 – ReduceValuesIterator ⾏行行