Unit 3 writable collections

There are six Writable collection types in the org.apache.hadoop.io
package:
1. Array Writable,
2. ArrayPrimitiveWritable,
3. TwoDArrayWritable,
4. MapWritable,
5. SortedMapWritable,
6. EnumSetWritable.
 ArrayWritable and TwoDArrayWritable are Writable implementations for arrays and
two-dimensional arrays (array of arrays) of Writable instances. All the elements of an
ArrayWritable or a TwoDArrayWritable must be instances of the same class, which is
specified at construction, as follows:
ArrayWritable writable = new ArrayWritable(Text.class);

In contexts where the Writable is defined by type, such as in SequenceFile
keys or values, or as input to MapReduce in general, you need to subclass
ArrayWritable (or TwoDArrayWritable, as appropriate) to set the type
statically. For example:
public class TextArrayWritable extends ArrayWritable
{
public TextArrayWritable()
{
super(Text.class);
}
}
ArrayWritable and TwoDArrayWritable both have get() and set()
methods, as well as a toArray() method, which creates a shallow
copy of the array.

ArrayPrimitiveWritable is a wrapper for arrays of Java primitives. The
component type is detected when you call set(), so there is no need to subclass
to set the type. MapWritable and SortedMapWritable are implementations of
java.util.Map and java.util.SortedMap, respectively. Here’s a demonstration of
using a MapWritable with different types for keys and values:
MapWritable src = new MapWritable();
src.put(new IntWritable(1), new Text("cat"));
src.put(new VIntWritable(2), new LongWritable(163));
MapWritable dest = new MapWritable();
WritableUtils.cloneInto(dest, src);
assertThat((Text) dest.get(new IntWritable(1)), is(new Text("cat")));
assertThat((LongWritable) dest.get(new VIntWritable(2)), is(new
LongWritable(163)));

 Writable collection implementations for sets and lists.
A general set can be emulated by using a MapWritable
(or a SortedMapWritable for a sorted set), with
NullWritable values. There is also EnumSetWritable for
sets of enum types. For lists of a single type of Writable,
ArrayWritable is adequate, but to store different types of
Writable in a single list, you can use GenericWritable to
wrap the elements in an ArrayWritable

 Hadoop comes with a useful set of Writable implementations that
serve most purposes; however, on occasion, you may need to write
your own custom implementation. nWith a custom Writable, you have
full control over the binary representation and the sort order.
Because Writables are at the heart of the MapReduce data path,
tuning the binary representation can have a significant effect on
performance. To demonstrate how to create a custom Writable, we
shall write an implementation that represents a pair of strings, called
TextPair. The basic implementation is shown in following Example.
Implementing a Custom Writable

import java.io.*;
import org.apache.hadoop.io.*;
public class TextPair implements
WritableComparable<TextPair>
{
private Text first;
private Text second;
public TextPair() {
set(new Text(), new Text());
}
public TextPair(String first, String second)
{
set(new Text(first), new Text(second));
}
public TextPair(Text first, Text second)
{
set(first, second);
}
public void set(Text first, Text second)
{this.first = first;
this.second = second; }
public Text getFirst() {
return first; }
public Text getSecond() {
return second; }
@Override
public void write(DataOutput out) throws
IOException {
first.write(out);
second.write(out); }
@Override
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in); }
@Override
public int hashCode() {
return first.hashCode() * 163 + second.hashCode(); }
@Override
public boolean equals(Object o) {
if (o instanceof TextPair) {
TextPair tp = (TextPair) o;
return first.equals(tp.first) && second.equals(tp.second); }
return false; }
@Override
public String toString() {
return first + "
t" + second;
}
@Override
public int compareTo(TextPair tp) {
int cmp = first.compareTo(tp.first);
if (cmp != 0) {
return cmp; }
return second.compareTo(tp.second);
}
}
A Writable implementation
that stores a pair of Text
objects

The first part of the implementation is straightforward: there are two Text instance
variables, first and second, and associated constructors, getters, and setters. All
Writable implementations must have a default constructor so that the MapReduce
framework can instantiate them, then populate their fields by calling readFields().
TextPair’s write() method serializes each Text object in turn to the output stream, by
delegating to the Text objects themselves. Similarly, readFields() deserializes the
bytes from the input stream by delegating to each Text object. The DataOutput and
DataInput interfaces have a rich set of methods for serializing and deserializing Java
Primitives.
Just as you would for any value object you write in Java, you should override the
hashCode(), equals(), and toString() methods from java.lang.Object. The hash Code()
method is used by the HashPartitioner (the default partitioner in MapReduce) to
choose a reduce partition, so you should make sure that you write a good hash
function that mixes well to ensure reduce partitions are of a similar size.
If you ever plan to use your custom Writable with TextOutputFormat, then you must
implement its toString() method. TextOutputFormat calls toString() on keys and values
for their output representation. For TextPair, we write the underlying Text objects as
strings separated by a tab character.
TextPair is an implementation of WritableComparable, so it provides an
implementation of the compareTo() method that imposes the ordering you would
expect: it sorts by the first string followed by the second.

Unit 3 writable collections

More Related Content

What's hot (20)

Similar to Unit 3 writable collections (20)

More from vishal choudhary (20)

Recently uploaded (20)

Unit 3 writable collections