map reduce 4..............................

2
Combiners
• When people think of map reduce, or dbs, they think
how to enhance the process by reduce data network
traffic.
• Increase the Map-Reduce processing efficiency using
the combiners.
• It is basically a function that does what the reducer
does, but on mapper side.

5
Combiners Motivation
• Moving data between nodes is one of the bottlenecks
in distributed systems.
• We need to find always solutions to reduce the
amount of data movement in the network.
• In most cases , mappers produce large amounts of
intermediate data passed on to the reducers for
further processing. Which leads to enormous network
congestion.
• One of the solutions is to reduce the mapper output
using compiners “mini reducers”

How combiners work
6
• The combiner must implement the Reducer interface’s reduce() method.
• The combiner process on each map output key & value (same as reducer).
• Combiner runs on the map side.
• The output of the combiner passed to the reducer.
• Reducers maybe used as combiners if the OP is commutative and associative.
• Combiner and reducer are often identical
• Combiners and reducer must have identical input and output data types.
• The operation must be commutative and associative.

Associative and commutative
• The commutative property states that you can move numbers around and you
still arrive at the same answer.
• a+b = b+a
• 1+2 =2+1
• A*B = B*A
• 1*2=2*1
• The associative property states that you can regroup numbers and you will get
the same answer.
• (a+b)+c=a+(b+c)
• (1+2)+3=1+(2+3)
• (A*B)*C=A*(B*C)
• (1*2)*3=1*(2*3)
7

With and without combiner
8
Mapper 1
Mapper 2
(the,1)
(the,1)
(the,1)
(the,1)
(the,1)
Sum(the ,1,1,1
)
Sum(the ,1,1)
Sum(the ,1,1,1,1,1)
Sum(the ,3,2
)
(The,5)
(The,5)

With and without combiner
9
Mapper 1
Mapper 2
(the,0.3
)
(the,0.4
)
(the,0.7
)
(the,0.5)
(the,0.6)
AVG(the ,0.3,0.4,0.
7)
AVG(the ,0.5,0.6)
AVG(the ,0.3,0.4,0.7,0.5,0
.6)
AVG(the ,0.467,0.5
5)
(The,0.5085)
(The,0.5)

Inverted index using map
reduce
11

10/22/2024 PRESENTATION TITLE 12
Map output
<The@file1, 1>
<Today@file1, 1>
<Today@file1, 1>
<Today@file1, 1>
<The@file2, 1>
<Today@file2, 1>

Combiner input &
output
<The@file1, (1)>
<Today@file1, (1,1,1)>
<The@file2, (1)>
<Today@file2, (1)>
<The file1:1>
<Today file1:3>
<The file2:1>
<Today file2:1>

Reducer input &
output
<The file1: 1, file2: 1 >
<Today file1:3, file2:1 >
<The 2 , (file1:1,file2:1)>
<Today 2, (file1:3,file2:1)>

map reduce 4..............................

More Related Content

Similar to map reduce 4.............................. (20)

Recently uploaded (20)

map reduce 4..............................

Editor's Notes