Repartition join in mapreduce
Download as PPTX, PDF0 likes1,991 views
This document discusses reduce-side joins in Hadoop. Reduce-side joins are performed by joining datasets in the reduce phase based on a join key. The data is organized by Hadoop to send identical keys to the same reducer, making reduce-side joins straightforward to implement. However, performance can suffer as all data is shuffled across the network. The document outlines the steps to perform a reduce-side join which include tagging each dataset with an identity, creating a composite key, partitioning and sorting the data, then joining in the reducer based on the key.
![1. Map output key
The key will be the empNo as it is the join key for the datasets employee and salary
[Implementation: in the mapper]
2. Tagging the data with the dataset identity
Add an attribute called srcIndex to tag the identity of the data (1=employee,
2=salary)
[Implementation: in the mapper]
3. Discarding unwanted atributes
[Implementation: in the mapper]
4. Composite key
Make the map output key a composite of empNo and srcIndex
[Implementation: create custom writable]
5. Partitioner
Partition the data on natural key of empNo
[Implementation: create custom partitioner class]
----ï continue
âą](https://image.slidesharecdn.com/psljtd7ytjgr5kshhh5c-signature-c40cf8dbee240b4fc5155d0586548e61735d77ca61cba18dffe996f7d5242bb7-poli-140925054638-phpapp02/85/Repartition-join-in-mapreduce-7-320.jpg)
![6. Sorting
Sort the data on empNo first, and then source index
[Implementation: create custom sorting comparator
class]
7. Grouping
Group the data based on natural key
[Implementation: create custom grouping comparator
class]
8. Joining
Iterate through the values for a key and complete the
join for employee and salary data.
[Implementation: in the reducer]](https://image.slidesharecdn.com/psljtd7ytjgr5kshhh5c-signature-c40cf8dbee240b4fc5155d0586548e61735d77ca61cba18dffe996f7d5242bb7-poli-140925054638-phpapp02/85/Repartition-join-in-mapreduce-8-320.jpg)
Repartition join in mapreduce
- 3. âą What is Reduce-side join? âą Steps used to join the datasets in Reduce-side join âą Sample datasets used in this project âą Scenario flow âą Practical demonstration of Reduce-side join
- 4. âą Joins of datasets done in the reduce phase based on join key are called reduce side joins. Reduce-side joins are the easiest to implement âą What makes reduce-side joins straight forward is the fact that Hadoop sends identical keys to the same reducer, so by default the data is organized for us. âą To perform the join, we simply need to cache a key and compare it to incoming keys. As long as the keys match, we can join the values from the corresponding keys. âą The trade off with reduce-side joins is performance, since all of the data is shuffled across the network
- 5. âą The key of the map output, of datasets being joined, has to be the join key - so they reach the same reducer âą Each dataset has to be tagged with its identity, in the mapper- to help differentiate between the datasets in the reducer, so they can be processed accordingly. âą In each reducer, the data values from both datasets, for keys assigned to the reducer, are available, to be processed as required. âą A secondary sort needs to be done to ensure the ordering of the values sent to the reducer âą If the input files are of different formats, we would need separate mappers, and we would need to use MultipleInputs class in the driver to add the inputs and associate the specific mapper to the same.
- 7. 1. Map output key The key will be the empNo as it is the join key for the datasets employee and salary [Implementation: in the mapper] 2. Tagging the data with the dataset identity Add an attribute called srcIndex to tag the identity of the data (1=employee, 2=salary) [Implementation: in the mapper] 3. Discarding unwanted atributes [Implementation: in the mapper] 4. Composite key Make the map output key a composite of empNo and srcIndex [Implementation: create custom writable] 5. Partitioner Partition the data on natural key of empNo [Implementation: create custom partitioner class] ----ï continue âą
- 8. 6. Sorting Sort the data on empNo first, and then source index [Implementation: create custom sorting comparator class] 7. Grouping Group the data based on natural key [Implementation: create custom grouping comparator class] 8. Joining Iterate through the values for a key and complete the join for employee and salary data. [Implementation: in the reducer]