BIG DATA ANALYTICS
- 1. UNDERSTANDING INPUTS AND OUTPUTS IN MAPREDUCE By, S.Subha Thilagam.
- 2. Your data might be XML files sitting behind a number of FTP servers, text log files sitting on a central web server, or Lucene indexes1 in HDFS. How does MapReduce support reading and writing to these different serialization structures across the various storage mechanisms? You’ll need to know the answer in order to support a specific serialization format.
- 4. The two classes that support data input in MapReduce are InputFormat and Record- Reader. The InputFormat class is consulted to determine how the input data should be partitioned for the map tasks, and the RecordReader performs the reading of data from the inputs.
- 5. Every job in MapReduce must define its inputs according to contracts specified in the InputFormat abstract class. InputFormat implementers must fulfill three contracts: first, they describe type information for map input keys and values; next, they specify how the input data should be partitioned; and finally, they indicate the RecordReader instance that should read the data from source.
- 6. The RecordReader class is used by MapReduce in the map tasks to read data from an input split and provide each record in the form of a key/value pair for use by mappers. A task is commonly created for each input split, and each task has a single RecordReader that’s responsible for reading the data for that input split.
- 7. MapReduce uses a similar process for supporting output data as it does for input data.Two classes must exist, an OutputFormat and a RecordWriter. The OutputFormat performs some basic validation of the data sink properties, and the RecordWriter writes each reducer output to the data sink.
- 8. Much like the InputFormat class, the OutputFormat class, as shown in figure 3.5, defines the contracts that implementers must fulfill, including checking the information related to the job output, providing a RecordWriter, and specifying an output committer, which allows writes to be staged and then made “permanent” upon task and/or job success.
- 9. You’ll use the RecordWriter to write the reducer outputs to the destination data sink. It’s a simple class.