SDEC2011 Essentials of Mahout

Conﬁdential, for personal use only. All original content copyright owned by Treasury of Ideas LLC.
Copyright for all other & referenced work is retained by their respective owners.

Essentials of Mahout
Mastering Hadoop Map-reduce for Data Analysis

Shashank Tiwari
blog: shanky.org | twitter: @tshanky
st@treasuryoﬁdeas.com


What is Apache Mahout?

• A scalable machine learning infrastructure

• Built on top of Hadoop MapReduce

• Currently supports:

• Clustering, classiﬁcation, and collaborative ﬁltering, etc...


A Little History

• Founded by folks active in the Lucene community

• Inspired by work at Stanford: “Map-Reduce for Machine Learning on
Multicore” -- http://www.cs.stanford.edu/people/ang/papers/nips06-
mapreducemulticore.pdf


Project Goal

• Create a community driven scalable and robust machine learning
infrastructure

• Leverage Hadoop for parallel processing and scalability

• Provide an abstraction on top of Hadoop so the machine-learning users are
not concerned with the map and reduce primitives when they build their
solutions.


Supported Algorithms

• Collaborative Filtering

• User and Item based recommenders

• K-Means, Fuzzy K-Means clustering

• Mean Shift clustering

• Dirichlet process clustering


More Supported Algorithms

• Latent Dirichlet Allocation

• Singular value decomposition

• Parallel Frequent Pattern mining

• Complementary Naive Bayes classiﬁer

• Random forest decision tree based classiﬁer

• ...and growing


Focus Areas

• Collaborative Filtering

• Clustering

• Classiﬁcation


Build and Install

• Required Software:

• Java 1.6.x

• Maven 2.0.11+

• Get source: svn co http://svn.apache.org/repos/asf/mahout/trunk mahout

• Compile & install core & examples: mvn install

• Alternatively, individually mvn compile, mvn package, and mvn install


Recommendation Examples

• mvn -q exec:java -
Dexec.mainClass="org.apache.mahout.cf.taste.example.grouplens.Group
LensRecommenderEvaluatorRunner" -Dexec.args="-i /Users/tshanky/
workspace/hadoop_workspace/grouplens/ratings.dat"

• https://cwiki.apache.org/conﬂuence/display/MAHOUT/
RecommendationExamples


Common Use Cases

• Shopping: Amazon, Netflix

• Who to follow/friend: Twitter/Facebook

• Web resource classification, spam filtering, financial markets pattern
recognition, classification


Collaborative Filtering Basis

• User-based: recommend items by ﬁnding similar users. User preferences
keep changing so this method poses challenges.

• Item-based: calculate similarity between items and make
recommendations. Usually items don’t change much so the method is
often reliable.

• Slope-one: fast and efﬁcient item based recommendation when user
ratings are more than boolean yes/no, like/dislike.

• Model-based: provide recommendation on the basis of developing a
model of users and their ratings.


Clustering Basis

• Clustering algorithms also use the notion of similarity to group similar
items into a cluster.

• Both Collaborative ﬁltering and clustering use the notion of a distance,
which could be calculated using a number of different techniques.

• Example: Euclidean distance,


Mahout Taste Framework

• Taste Collaborative Filtering:

• Taste is an open source project for CF started by Sean Owen on
SourceForge and donated to Mahout in 2008.

• Has been applied to a number of different data sets successfully.


Mahout Taste Framework

• Taste Collaborative Filtering:

• Taste is an open source project for CF started by Sean Owen on
SourceForge and donated to Mahout in 2008.

• Has been applied to a number of different data sets successfully.

• Mahout supports building recommendation engines primarily basis the Taste
library.

• The library supports both user-based and item-based recommendations.

• Can be used with Java or over RESTful web-service endpoints.


Taste Framework : Primary Classes

• DataModel: Model for Users, Items, and Preferences

• UserSimilarity: Interface deﬁning the similarity between two users

• ItemSimilarity: Interface deﬁning the similarity between two items

• Recommender: Interface for providing recommendations

• UserNeighborhood: Interface for computing a neighborhood of similar
users. These are used by the Recommenders.


Taste Framework : Online vs Ofﬂine

• Can do online recommendations for a few thousand data sets.

• Leverages Hadoop for ofﬂine recommendation calculations on large data
sets.


Understanding the Group Lens Implementation

• Provide an insight into a sample Mahout Taste Framework Implementation.

• Uses the publicly available data set

• Part of the distribution so you can analyze it, modify it, and use it as an
inspiration for your own implementation

• Easy to follow example


Group Lens Implementation Source

• GroupLensDataModel.java

• GroupLensRecommender.java

• GroupLensRecommenderBuilder.java

• GroupLensRecommenderEvaluatorRunner.java


Group Lens Runner -- evaluator

• Instantiates an evaluator:

• RecommenderEvaluator evaluator = new
AverageAbsoluteDifferenceRecommenderEvaluator();

• a “mean average error” algorithm

• Parses input parameters:

• File ratingsFile = TasteOptionParser.getRatings(args);


Group Lens Runner -- data model

• Parses a colon delimiter pattern ﬁle:

• DataModel model = ratingsFile == null ? new GroupLensDataModel() :
new GroupLensDataModel(ratingsFile);

Group Lens Runner -- evaluate with

recommendation builder

• evaluates using GroupLensRecommender

• double evaluation = evaluator.evaluate(new
GroupLensRecommenderBuilder(), null, model, 0.9, 0.3);


Questions?

• blog: shanky.org | twitter: @tshanky

• st@treasuryoﬁdeas.com

SDEC2011 Essentials of Mahout

More Related Content

Similar to SDEC2011 Essentials of Mahout (20)

More from Korea Sdec (14)

Recently uploaded (20)

SDEC2011 Essentials of Mahout