Hadoop Puzzlers

1
Hadoop Puzzlers
Aaron Myers & Daniel Templeton
Cloudera, Inc.

2
Your Hosts
Aaron “ATM” Myers
• AKA “Cash Money”
• Software Engineer
• Apache Hadoop
Committer
Daniel Templeton
• Certification Developer
• Crusty, old HPC guy
• Likes Perl
©2014 Cloudera, Inc. All rights reserved.2

3
What is a Hadoop Puzzler
• Shameless knockoff of Josh Bloch’s Java Puzzlers talks
• We’ll walk through a puzzle
• You vote on the answer
• We all learn a valuable lesson

4 ©2014 Cloudera, Inc. All rights reserved.4
Let’s try it, OK?

5
An Easy One
public class MaxMap
extends Mapper<LongWritable,
Text,Text,IntWritable> {
Text k = new Text();
IntWritable v = new IntWritable();
protected void map(LongWritable key,
Text val, Context c) … {
String[] parts =
val.toString().split(",");
k.set(parts[0]);
v.set(Integer.parseInt(parts[1]));
c.write(k, v);
} }
public class MaxReduce
extends Reducer<Text,IntWritable,
Text,IntWritable> {
protected void reduce(Text key,
Iterable<IntWritable> values,
Context c) … {
IntWritable max =
new IntWritable(0);
for (IntWritable v: values)
if (v.get() > max.get())
max = v;
c.write(key, max);
} }

6
An Easy One
The data:
A,1
A,5
A,3
The results:
a) A 5
b) A 1
c) A 3
d) The job fails

7
An Easy One
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
IntWritable max =
new IntWritable(0);
max = v;
c.write(key, max);
} }
A 1
A 5
A 3

8
An Easy One
The data:
A,1
A,5
A,3
The results:
a) A 5
b) A 1
c) A 3
d) The job fails

9
An Easy One (Answer)
The data:
A,1
A,5
A,3
The results:
a) A 5
b) A 1
c) A 3
d) The job fails

10
An Easy One (Problem)
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
IntWritable max =
new IntWritable(0);
max = v;
c.write(key, max);
} }

11
An Easy One (Moral)
• MapReduce reuses Writables whenever it can
• That includes while iterating through the values
• Always be careful to only store the value instead of
the Writable!

12
A Sinking Feeling
public class AsyncSubmit
extends Configured
implements Tool {
public static void main(String[] args)
throws Exception {
int ret = ToolRunner.run(
new Configuration(),
new AsyncSubmit(), args);
System.exit(ret);
}
public int run(String[] args)
throws Exception {
Job job = Job.getInstance(getConf());
job.setNumReduceTasks(0);
FileInputFormat.addInputPath(job,
new Path(args[0]));
FileOutputFormat.setOutputPath(job,
new Path(args[1]));
job.waitForCompletion(false);
return job.isComplete() ? 1 : 0;
} }

13
A Sinking Feeling
The data:
The complete works of
William Shakespeare
The results:
a) Fails to compile
b) The job fails
c) Exits with 0
d) Exits with 1

14
A Sinking Feeling
extends Configured
implements Tool {
throws Exception {
System.exit(ret);
}
throws Exception {
new Path(args[0]));
new Path(args[1]));
} }
William Shakespeare

15
A Sinking Feeling
The data:
William Shakespeare
The results:
a) Fails to compile
b) The job fails
c) Exits with 0
d) Exits with 1

16
A Sinking Feeling (Answer)
The data:
William Shakespeare
The results:
a) Fails to compile
b) The job fails
c) Exits with 0
d) Exits with 1

17
A Sinking Feeling (Problem)
extends Configured
implements Tool {
throws Exception {
System.exit(ret);
}
throws Exception {
new Path(args[0]));
new Path(args[1]));
} }

18
A Sinking Job (Moral)
• Read the API docs!
• Sometimes the obvious meanings of methods and
parameters aren’t correct
• Parameter for waitForCompletion() controls whether
status output is printed
• Driver does wait for job to exit but does not print all the job
status information

19
Do-over
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
public class MaxReduceRedux
extends Reducer<Text,Text,
Text,IntWritable> {
Context c) … {
int max = 0;
if (v.get() > max)
max = v.get();
c.write(key, new IntWritable(max));
} }

20
Do-over
The data:
A,1
A,5
The results:
a) A 5
b) A 1
c) A 1
A 5
d) The job fails

21
Do-over
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
int max = 0;
if (v.get() > max)
max = v.get();
} }
A 1
A 5

22
Do-over
The data:
A,1
A,5
The results:
a) A 5
b) A 1
c) A 1
A 5
d) The job fails

23
Do-over (Answer)
The data:
A,1
A,5
The results:
a) A 5
b) A 1
c) A 1
A 5
d) The job fails

24
Do-over (Problem)
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
int max = 0;
if (v.get() > max)
max = v.get();
} }

25
Do-over (Moral)
• Mismatched signatures can lead to unexpected
behaviors because of exposed base class method
implementations
• ALWAYS use @Override!

26
Joining Forces
hive> DESCRIBE table1;
OK
id int
phone string
state string
Time taken: 0.236 seconds
OK
id int
city string
state string
hive> CREATE TABLE table3 AS SELECT
table2.*,table1.phone,table1.state
AS s FROM table1 JOIN table2 ON
(table1.id == table2.id);
…
hive> EXPORT TABLE table3 TO
'/user/cloudera/table3.csv';
…
hive> exit
$ hadoop fs –cat table3.csv |
head -1 | tr , 'n' | wc –l

27
Joining Forces
The data:
hive> SELECT * FROM table1;
OK
1 6506506500 CA
2 2282282280 MS
OK
1 Palo Alto CA
2 Gautier MS
The results:
a) 5
b) 4
c) 1
d) The join fails

28
Joining Forces
OK
id int
phone string
state string
OK
id int
city string
state string
…
…
hive> exit
head -1 | tr , 'n' | wc –l
1 6506506500 CA
2 2282282280 MS
1 Palo Alto CA
2 Gautier MS

29
Joining Forces
The data:
OK
1 6506506500 CA
2 2282282280 MS
OK
1 Palo Alto CA
2 Gautier MS
The results:
a) 5
b) 4
c) 1
d) The join fails

30
Joining Forces (Answer)
The data:
OK
1 6506506500 CA
2 2282282280 MS
OK
1 Palo Alto CA
2 Gautier MS
The results:
a) 5
b) 4
c) 1
d) The join fails

31
Joining Forces (Problem)
OK
id int
phone string
state string
OK
id int
city string
state string
…
…
hive> exit
head -1 | tr , 'n' | wc –l

32
Joining Forces (Moral)
• Hive’s default delimiter is 0x01 (CTRL-A)
• Easy to assume export will use a sane delimiter – it
doesn’t
• Incidentally, Hive’s join rules are pretty sane and work
as you’d expect

33
Close Enough
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
public class Top20Reduce
Text,IntWritable> {
Context c) … {
float max = 0.0f;
if (v.get() > max) max = v.get();
max *= 0.8f;
if (v.get() >= max)
c.write(key, v);
} }

34
Close Enough
The data:
A,1
A,5
A,4
The results:
a)
b) A 5
c) A 5
A 4
d) The job fails

35
Close Enough
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
float max = 0.0f;
max *= 0.8f;
if (v.get() >= max)
c.write(key, v);
} }
A 1
A 5
A 4

36
Close Enough
The data:
A,1
A,5
A,4
The results:
a)
b) A 5
c) A 5
A 4
d) The job fails

37
Close Enough (Answer)
The data:
A,1
A,5
A,4
The results:
a)
b) A 5
c) A 5
A 4
d) The job fails

38
Close Enough (Problem)
public class MaxMap
String[] parts =
k.set(parts[0]);
c.write(k, v);
} }
Text,IntWritable> {
Context c) … {
float max = 0.0f;
max *= 0.8f;
if (v.get() >= max)
c.write(key, v);
} }

39
Close Enough (Moral)
• For scalability reasons, the values iterable is
single-shot
• Subsequent iterators iterate over an empty collection
• Store values (not Writables!) in the first pass
• Better yet, restructure the logic to avoid storing all
values in memory

40
Overbyte
public class MinLineMap
Text,Text,Text> {
Text value, Context c) … {
String val = value.toString();
k.set(val.substring(0, 1));
c.write(k, value);
} }
public class MinLineReduce
Text,IntWritable> {
Iterable<Text> values,
Context c) … {
int min = Integer.MAX_VALUE;
for (Text v: values)
if (v.getBytes().length < min)
min = v.getBytes().length;
c.write(key, new IntWritable(min));
} }

41
Overbyte
The data:
Hadoop
Spark
Hive
Sqoop2
The results:
a) H 4
S 5
b) H 6
S 5
c) H 6
S 6
d) The job fails

42
Overbyte
Text,Text,Text> {
c.write(k, value);
} }
Text,IntWritable> {
Context c) … {
} }
Hadoop
Spark
Hive
Sqoop2

43
Overbyte
The data:
Hadoop
Spark
Hive
Sqoop2
The results:
a) H 4
S 5
b) H 6
S 5
c) H 6
S 6
d) The job fails

44
Overbyte (Answer)
The data:
Hadoop
Spark
Hive
Sqoop2
The results:
a) H 4
S 5
b) H 6
S 5
c) H 6
S 6
d) The job fails

45
Overbyte (Problem)
Text,Text,Text> {
c.write(k, value);
} }
Text,IntWritable> {
Context c) … {
} }

46
Overbyte (Moral)
• Writables get reused in loops
• In addition, Text.getBytes() reuses byte array
allocated by previous calls
• Net result is wrongness
• Text.getLength() is the correct way to get the length
of a Text.

47
What We Learned
• Beware of reuse of Writables
• Always use @Override so your compiler can help you
• Don’t assume you know what a method does because
of the name or parameters – read the docs!
• Sometimes scalability is inconvenient

48
One Closing Note
• Hadoop is still not easy
• Being good takes effort and experience
• Recognizing Hadoop talent can be hard
• Cloudera’s is working to make Hadoop talent easier to
recognize through certification
http://cloudera.com/content/cloudera/en/training/cert
ification.html

Hadoop Puzzlers

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to Hadoop Puzzlers (20)

More from Cloudera, Inc. (20)

Recently uploaded (20)

Hadoop Puzzlers