Introduction to Mahout given at Twin Cities HUG

1©MapR Technologies 2013- Confidential
Introduction to Mahout
And How To Build a Recommender

Me, Us
 Ted Dunning, Chief Application Architect, MapR
Committer PMC member, Mahout, Zookeeper, Drill
Bought the beer at the first HUG
 MapR
Distributes more open source components for Hadoop
Adds major technology for performance, HA, industry standard API’s
 Tonight
Hash tag - #tchug
See also - @ApacheMahout @ApacheDrill
@ted_dunning and @mapR

Sidebar on Drill
 Apache Drill
– SQL on Hadoop (and other things)
– Intended to solve problems for 1-5 years from now
Not the problems from 1-10 years ago
– Multiple levels of API supported
• SQL-2003
• Logical plan language (DAG in JSON)
• Physical plan language (DAG with push-down, exchange markers)
• Execution plan language (many DAG’s)
 Current state
– SQL 2003 support in place
– Logical plan interpreter useful for testing
– Value vectors near completion
– High performance RPC working

More on Drill
 Just completed OSCON workshop
 Workshop materials available shortly
– Extracted technology demonstrators
– Sample queries
 Send me email or tweet for more info

What’s Up?
 What is Mahout?
– Math library
– Clustering, classifiers, other stuff
 Recommendation
– Generalities
– Algorithm Specifics
– System Design
– Important things never mentioned
 Final thoughts

What is Mahout?
 “Scalable machine learning”
– not just Hadoop-oriented machine learning
– not entirely, that is. Just mostly.
 Components
– math library
– clustering
– classification
– decompositions
– recommendations

Mahout Math

Mahout Math
 Goals are
– basic linear algebra,
– and statistical sampling,
– and good clustering,
– decent speed,
– extensibility,
– especially for sparse data
 But not
– totally badass speed
– comprehensive set of algorithms
– optimization, root finders, quadrature

Matrices and Vectors
 At the core:
– DenseVector, RandomAccessSparseVector
– DenseMatrix, SparseRowMatrix
 Highly composable API
 Important ideas:
– view*, assign and aggregate
– iteration
m.viewDiagonal().assign(v)

Assign? View?
 Why assign?
– Copying is the major cost for naïve matrix packages
– In-place operations critical to reasonable performance
– Many kinds of updates required, so functional style very helpful
 Why view?
– In-place operations often required for blocks, rows, columns or diagonals
– With views, we need #assign + #views methods
– Without views, we need #assign x #views methods
 Synergies
– With both views and assign, many loops become single line

Assign
 Matrices
 Vectors
Matrix assign(double value);
Matrix assign(double[][] values);
Matrix assign(Matrix other);
Matrix assign(DoubleFunction f);
Matrix assign(Matrix other, DoubleDoubleFunction f);
Vector assign(double value);
Vector assign(double[] values);
Vector assign(Vector other);
Vector assign(DoubleFunction f);
Vector assign(Vector other, DoubleDoubleFunction f);
Vector assign(DoubleDoubleFunction f, double y);

Views
 Matrices
 Vectors
Matrix viewPart(int[] offset, int[] size);
Matrix viewPart(int row, int rlen, int col, int clen);
Vector viewRow(int row);
Vector viewColumn(int column);
Vector viewDiagonal();
Vector viewPart(int offset, int length);

Aggregates
 Matrices
 Vectors
double zSum();
double aggregate(
DoubleDoubleFunction reduce, DoubleFunction map);
double aggregate(Vector other,
DoubleDoubleFunction aggregator,
DoubleDoubleFunction combiner);
double zSum();
Vector aggregateRows(VectorFunction f);
Vector aggregateColumns(VectorFunction f);
double aggregate(DoubleDoubleFunction combiner,
DoubleFunction mapper);

Predefined Functions
 Many handy functions
ABS LOG2
ACOS NEGATE
ASIN RINT
ATAN SIGN
CEIL SIN
COS SQRT
EXP SQUARE
FLOOR SIGMOID
IDENTITY SIGMOIDGRADIENT
INV TAN
LOGARITHM

Examples
double alpha; a.assign(alpha);
a.assign(b, Functions.chain(
Functions.plus(beta),
Functions.times(alpha));
A =a
A =aB+ b

Sparse Optimizations
 DoubleDoubleFunction abstract properties
 And Vector properties
public boolean isLikeRightPlus();
public boolean isLikeLeftMult();
public boolean isLikeRightMult();
public boolean isLikeMult();
public boolean isCommutative();
public boolean isAssociative();
public boolean isAssociativeAndCommutative();
public boolean isDensifying();
public boolean isDense();
public boolean isSequentialAccess();
public double getLookupCost();
public double getIteratorAdvanceCost();
public boolean isAddConstantTime();

More Examples
 The trace of a matrix
 Set diagonal to zero
 Set diagonal to negative of row sums

Examples
m.viewDiagonal().zSum()

Examples
m.viewDiagonal().assign(0)

Examples
 Set diagonal to negative of row sums excluding the diagonal
m.viewDiagonal().assign(0)
Vector diag = m.viewDiagonal().assign(0);
diag.assign(m.rowSums().assign(Functions.MINUS));

Iteration
 Matrices are Iterable in Mahout
 Vectors are densely or sparsely iterable
// compute both row and columns sums in one pass
for (MatrixSlice row: m) {
rSums.set(row.index(), row.zSum());
cSums.assign(row, Functions.PLUS);
}
double entropy = 0;
for (Vector.Element e: v.nonZeroes()) {
entropy += e.get() * Math.log(e.get());
}

Random Sampling
 Samples from some type
 Lots of kinds
ChineseRestaurant Missing Normal
Empirical Multinomial PoissonSampler
IndianBuffet MultiNormal Sampler
public interface Sampler<T> {
T sample();
}
public abstract class AbstractSamplerFunction
extends DoubleFunction
implements Sampler<Double>

Clustering and Such
 Streaming k-means and ball k-means
– streaming reduces very large data to a cluster sketch
– ball k-means is a high quality k-means implementation
– the cluster sketch is also usable for other applications
– single machine threaded and map-reduce versions available
 SVD and friends
– stochastic SVD has in-memory, single machine out-of-core and map-reduce
versions
– good for reducing very large sparse matrices to tall skinny dense ones
 Spectral clustering
– based on SVD, allows massive dimensional clustering

Mahout Math Summary
 Matrices, Vectors
– views
– in-place assignment
– aggregations
– iterations
 Functions
– lots built-in
– cooperate with sparse vector optimizations
 Sampling
– abstract samplers
– samplers as functions
 Other stuff … clustering, SVD

Recommenders

Recommendations
 Often known as collaborative filtering
 Actors interact with items
– observe successful interaction
 We want to suggest additional successful interactions
 Observations inherently very sparse

The Big Ideas
 Cooccurrence is the core operation (and it is pretty simple)
 Cooccurrence can be extended to handle important new
capabilities
 Recommendation systems can be deployed ideally using search
technology

Examples of Recommendations
 Customers buying books (Linden et al)
 Web visitors rating music (Shardanand and Maes) or movies (Riedl,
et al), (Netflix)
 Internet radio listeners not skipping songs (Musicmatch)
 Internet video watchers watching >30 s (Veoh)
 Visibility in a map UI (new Google maps)

A simple recommendation architecture
 Look at the history of interactions
 Find significant item cooccurrence in user histories
 Use these cooccurring items as “indicators”
 For all indicators in user history, accumulate scores for related
items

Recommendation Basics
 History:
User Thing
1 3
2 4
3 4
2 3
3 2
1 1
2 1

 History as matrix:
 (t1, t3) cooccur 2 times,
 (t1, t4) once,
 (t2, t4) once,
 (t3, t4) once
t1 t2 t3 t4
u1 1 0 1 0
u2 1 0 1 1
u3 0 1 0 1

A Quick Simplification
 Users who do h
 Also do r
Ah
AT
Ah( )
AT
A( )h
User-centric recommendations
Item-centric recommendations

 Coocurrence
t1 t2 t3 t4
t1 2 0 2 1
t2 0 1 0 1
t3 2 0 1 1
t4 1 1 1 2

Problems with Raw Cooccurrence
 Very popular items co-occur with everything
– Welcome document
– Elevator music
 That isn’t interesting
– We want anomalous cooccurrence

 Coocurrence
t1 t2 t3 t4
t1 2 0 2 1
t2 0 1 0 1
t3 2 0 1 1
t4 1 1 1 2
t3 not t3
t1 2 1
not t1 1 1

Spot the Anomaly
 Root LLR is roughly like standard deviations
A not A
B 13 1000
not B 1000 100,000
A not A
B 1 0
not B 0 2
A not A
B 1 0
not B 0 10,000
A not A
B 10 0
not B 0 100,000
0.44 0.98
2.26 7.15

Threshold by Score
 Coocurrence
t1 t2 t3 t4
t1 2 0 2 1
t2 0 1 0 1
t3 2 0 1 1
t4 1 1 1 2

Threshold by Score
 Significant cooccurrence => Indicators
t1 t2 t3 t4
t1 1 0 0 1
t2 0 1 0 1
t3 0 0 1 1
t4 1 0 0 1

So Far, So Good
 Classic recommendation systems based on these approaches
– Musicmatch (ca 2000)
– Veoh Networks (ca 2005)
 Currently available in Mahout
– See RowSimilarityJob
 Very simple to deploy
– Compute indicators
– Store in search engine
– Works very well with enough data

What’s right
about this?

Virtues of Current State of the Art
 Lots of well publicized history
– Musicmatch, Veoh, Netflix, Amazon, Overstock
 Lots of support
– Mahout, commercial offerings like Myrrix
 Lots of existing code
– Mahout, commercial codes
 Proven track record
 Well socialized solution

What’s wrong
about this?

Problems for Recommenders
 Cold start
 Disjoint populations
 Long tail
 Multiple kinds of evidence (multi-modal recommendations)
– unstructured add-on data
– other transaction streams
– textual descriptions

What is this multi-modal stuff?
 But people don’t just do one thing
 One kind of behavior is useful for predicting other kinds
 Having a complete picture is important for accuracy
 What has the user said, viewed, clicked, closed, bought lately?

Example Multi-modal Inputs
 Overlap in restaurant visits is useful
 Big spender cues
 Cuisine as an indicator
 Review text as an indicator

Too Limited
 People do more than one kind of thing
 Different kinds of behaviors give different quality, quantity and
kind of information
 We don’t have to do co-occurrence
 We can do cross-occurrence
 Result is cross-recommendation

Heh?

For example
 Users enter queries (A)
– (actor = user, item=query)
 Users view videos (B)
– (actor = user, item=video)
 ATA gives query recommendation
– “did you mean to ask for”
 BTB gives video recommendation
– “you might like these videos”

The punch-line
 BTA recommends videos in response to a query
– (isn’t that a search engine?)
– (not quite, it doesn’t look at content or meta-data)

Real-life example
 Query: “Paco de Lucia”
 Conventional meta-data search results:
– “hombres del paco” times 400
– not much else
 Recommendation based search:
– Flamenco guitar and dancers
– Spanish and classical guitar
– Van Halen doing a classical/flamenco riff

Real-life example

Hypothetical Example
 Want a navigational ontology?
 Just put labels on a web page with traffic
– This gives A = users x label clicks
 Remember viewing history
– This gives B = users x items
 Cross recommend
– B’A = label to item mapping
 After several users click, results are whatever users think they
should be

Nice. But we
can do better?

Ausers
things

A1 A2
é
ë
ù
û
users
thing
type 1
thing
type 2

A1 A2
é
ë
ù
û
T
A1 A2
é
ë
ù
û=
A1
T
A2
T
é
ë
ê
ê
ù
û
ú
ú
A1 A2
é
ë
ù
û
=
A1
T
A1 A1
T
A2
AT
2A1 AT
2A2
é
ë
ê
ê
ù
û
ú
ú
r1
r2
é
ë
ê
ê
ù
û
ú
ú
=
A1
T
A1 A1
T
A2
AT
2A1 AT
2A2
é
ë
ê
ê
ù
û
ú
ú
h1
h2
é
ë
ê
ê
ù
û
ú
ú
r1 = A1
T
A1 A1
T
A2
é
ëê
ù
ûú
h1
h2
é
ë
ê
ê
ù
û
ú
ú

Summary
 Input: Multiple kinds of behavior on one set of things
 Output: Recommendations for one kind of behavior with a
different set of things
 Cross recommendation is a special case

Now again, without
the scary math

Input Data
 User transactions
– user id, merchant id
– SIC code, amount
– Descriptions, cuisine, …
 Offer transactions
– user id, offer id
– vendor id, merchant id’s,
– offers, views, accepts

Input Data
 User transactions
– user id, merchant id
– SIC code, amount
– Descriptions, cuisine, …
 Offer transactions
– user id, offer id
– vendor id, merchant id’s,
– offers, views, accepts
 Derived user data
– merchant id’s
– anomalous descriptor terms
– offer & vendor id’s
 Derived merchant data
– local top40
– SIC code
– vendor code
– amount distribution

Cross-recommendation
 Per merchant indicators
– merchant id’s
– chain id’s
– SIC codes
– indicator terms from text
– offer vendor id’s
 Computed by finding anomalous (indicator => merchant) rates

How can we deploy
this?

Search-based Recommendations
 Sample document
– Merchant Id
– Field for text description
– Phone
– Address
– Location

 Sample document
– Merchant Id
– Phone
– Address
– Location
– Indicator merchant id’s
– Indicator industry (SIC) id’s
– Indicator offers
– Indicator text
– Local top40

 Sample document
– Merchant Id
– Phone
– Address
– Location
– Indicator text
– Local top40
 Sample query
– Current location
– Recent merchant descriptions
– Recent merchant id’s
– Recent SIC codes
– Recent accepted offers
– Local top40

 Sample document
– Merchant Id
– Phone
– Address
– Location
– Indicator text
– Local top40
 Sample query
– Current location
– Recent merchant descriptions
– Recent merchant id’s
– Recent SIC codes
– Recent accepted offers
– Local top40
Original data
and meta-data
Derived from cooccurrence
and cross-occurrence
analysis
Recommendation
query

SolR
Indexer
SolR
Indexer
Solr
indexing
Cooccurrence
(Mahout)
Item meta-
data
Index
shards
Complete
history
Analyze with Map-Reduce

SolR
Indexer
SolR
Indexer
Solr
search
Web tier
Item meta-
data
Index
shards
User
history
Deploy with Conventional Search System

Objective Results
 At a very large credit card company
 History is all transactions
 Development time to minimal viable product about 4 months
 General release 2-3 months later
 Search-based recs at or equal in quality to other techniques

 Contact:
– tdunning@maprtech.com
– @ted_dunning
– @apachemahout
– @user-subscribe@mahout.apache.org
 Slides and such
http://www.slideshare.net/tdunning
 Hash tags: #mapr #apachemahout #recommendations

Introduction to Mahout given at Twin Cities HUG

More Related Content

What's hot (15)

Viewers also liked (9)

Similar to Introduction to Mahout given at Twin Cities HUG (20)

More from MapR Technologies (20)

Recently uploaded (20)

Introduction to Mahout given at Twin Cities HUG