Thoughts on MongoDB Analytics

ANALYTICS WITH MONGODB

ROGER BODAMER

BUT HOW ?

• These graphs are the end result of a process

• In
order get here there’s a few things you need to do and
explore

A WORD ON NON-NATIVE
APPROACHES
• Yes, you can

• map your document schema to a relational schema

• then export your data from MongoDB to a relational db

• and set up a cron job to do this every day

• then use your BI tool to map relational to “objects”

• and then Report and do Analytics

BUT THAT WOULD BE NO
FUN

• Analytics using Native Queries

•A simple process

PROCESS: NAIVE

• Take a sample document

• Develop query

• Put on chart

• Done !

• and a gold star from your boss !

PROCESS: REALITY
• Understand your schema
• multiple schema’s in single collection
• multiple collections / multiple data sources
• Iterate:
• deﬁne metric
• develop query and report on metrics
• understand and drill down or discard
• repeat
• Operationalize metrics: dashboard
• Dimensions
• Plotting

UNDERSTAND YOUR SCHEMA

{
"name" : "Mario",
"games" : [{"game" : "WoW",
"duration" : 130},
{"game" : "Tetris",
"duration" : 130}]
}

BUT ALSO:
• Schema’s can be Polymorphic

{
"name" : "Bob",
"location" : "us",
"games" : [{"game" : "WoW",
"duration" : 2910},
{"game" : "Tetris",
"duration" : 593}]
}

SO NOW WHAT ?
• Only report on common attributes

• probably missing the most recent / interesting data

SO NOW WHAT ?
• Write 2 programs, one for each schema

• 2 graphs / reports

• 2 programs writing to 1 graph (basically merging instance data in 2
places)

SO NOW WHAT ?

• Unify Schema

• deal with absent, null values

• translate(NULL, “EU”);

ITERATE

• total time and how many games people play in the us vs eu ?

QUERY
db.runCommand(
{ aggregate : "gamers", pipeline : [
{ $project : {

location : 1,

games: 1
}},
{ $unwind : "$games" },
{ $group : {
_id : { location : 1},

number_games: { $sum : 1 },
total_duration: {$sum : "$games.duration"}
}},
{ $project : {

_id : 0,
location : "$_id.location",

number_games : 1,
total_duration : 1
}}
]})

SIDEBAR: WRITING
AGGREGATION QUERIES
• Prepare Data
• Extract relevant properties from collection documents
• Unwind sub collection if its document is contributing to aggregation
• Aggregate data
• determine the key (_id) on which the aggregates should be done
• name aggregates
• Project Data
• For ﬁnal results

EXAMPLE
{
"name" : "Alice",
"location" : "us",
"games" : [{
"game" : "WoW",
"duration" : 200
}, {
"game" : "Tetris",
"duration" : 100
}]
}

PREPARE
• Only use location and games:

{ $project : {

location : 1,

games: 1
}}

• Unwind games as properties of its documents are aggregated
over:

{ $unwind : "$games" }

AGGREGATE DATA
• Aggregate on number of games (add 1 per game)
and total duration (add duration per game)
using location as key

{ $group : {
_id : { location : 1},

total_duration: {$sum : "$games.duration"}
}}

PROJECT
• Only show location and aggregates, do not show _id

{ $project : {

_id : 0,

number_games : 1,
total_duration : 1
}}

RESULT 1

• People spend a little more time playing in the US
• More games played in the EU

CHALLENGE 2

• Since
we found EU and US play similar amount and same
number of games, new challenge is:

• Lets
see what the distribution of different
games is the 2 locations

QUERY 2
db.runCommand(
{ $project : {

location : 1,

games : 1
}},
{ $project : {

location : 1,

game : "$games.game",
duration : "$games.duration"
}},
{ $group : {
_id : { location: "$location", game: "$game"},

total_duration: {$sum : "$duration"}
}},
{ $project : {
_id : 0,
game : "$_id.game",

number_games : 1,
total_duration : 1
}}
]})

QUERY 2
db.runCommand(
{ $project : {

location : 1, location, games

games : 1
}},
{ $project : {

location : 1,

game : "$games.game",
}},
{ $group : {

}},
{ $project : {
_id : 0,
game : "$_id.game",

number_games : 1,
total_duration : 1
}}
]})

QUERY 2
db.runCommand(
{ $project : {


games : 1
}},
{ $project : {

location : 1,

game : "$games.game", location, game, duration
}},
{ $group : {

}},
{ $project : {
_id : 0,
game : "$_id.game",

number_games : 1,
total_duration : 1
}}
]})

QUERY 2
db.runCommand(
{ $project : {


games : 1
}},
{ $project : {

location : 1,

}},
{ $group : {
_id : { location: "$location", game: "$game"}, key: aggregate on location and game

}},
{ $project : {
_id : 0,
game : "$_id.game",

number_games : 1,
total_duration : 1
}}
]})

QUERY 2
db.runCommand(
{ $project : {


games : 1
}},
{ $project : {

location : 1,

}},
{ $group : {
_id : { location: "$location", game: "$game"}, key: aggregate on location and game

}},
{ $project : {
_id : 0,
location : "$_id.location", project: location, game, total(#games), sum(duration)
game : "$_id.game",

number_games : 1,
total_duration : 1
}}
]})

RESULT 2

Count: EU - WoW, US Tetris
EU spends more time on WoW, US it’s more
evenly spread

CHALLENGE 3:

• How do I compare Bob to everyone else in the EU ?

QUERY

•2 aggregations happening at same time:

•1 by user

•1 by location

• This query needs to be broken up in several queries

• Fairly complex

• Currently easiest to process in Ruby/Java/Python/...

db.runCommand( db.runCommand(
{ aggregate : "gamers", pipeline : [ { aggregate : "gamers", pipeline : [
{ $project : { { $project : {
name : 1,

location : 1,

location : 1,

games : 1

games : 1 }},
}}, { $unwind : "$games" },
{ $unwind : "$games" }, { $project : {
{ $project : { location : 1,

name: 1, duration : "$games.duration"
location : 1, }},

game : "$games.game", { $group : {
duration : "$games.duration" _id : { location: 1},
}}, total_duration: {$sum :
{ $group : { "$duration"}
_id : { location: "$location", name: "$name", game: }},
"$game"}, { $project : {

name : "$_id.location",
}}, _id : 0,
{ $project : { total_duration : 1

name : "$_id.name", }}
_id : 0, ]})
game : "$_id.game",
total_duration : 1
}}
]})

RESULT 3

• Bob plays >20% WoW in comparison to the Europeans, but
plays 200% more Tetris

A NOTE ON QUERIES

• There’s no notion of a declared schema

• The augmented scheme is coded in queries

• Reuse is very hard, happens at a query language

DIMENSIONS
• Most questions / graphs have a dimension

• Time, Geo

• Categories

• Relative: what’s X’s contribution of revenue to total

• Youwill need to be able to pass in dimensions as a
predicate for your queries

• or cache result and post process client-side

A WORD ON RENDERING
GRAPHS / REPORTS
• Several libraries available for ruby / python / java

• Gruff, Scruffy, StockCharts, D3, JRafael, JQuery Vizualize,
MooCharts, etc, etc.

• Also some services: John Nunemakers work (http://
get.gaug.es/)

• But Basically:

• you know how to program, right !

REVIEW
• Understand your schema
• multiple schema’s in single collection
• multiple collections / multiple data sources
• Iterate:
• deﬁne metric
• develop query and report on metrics
• understand and drill down or discard
• repeat
• Operationalize metrics: dashboard
• Dimensions
• Plotting

PUNCHLINES

• We have described a software engineering process

• but requirements will be very ﬂuid

• When you know how to write ruby / java / python etc. - life is
good

• If you’re a business analyst you have a problem

• better be BFF with some engineer :)

PLUG

• We’ve been working on a declarative analytics product

• (initially) uses Excel as its presentation layer

• Reach out to me if you’re interested

@rogerb
roger@norellan.com

Thoughts on MongoDB Analytics

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Thoughts on MongoDB Analytics (20)

More from rogerbodamer (6)

Recently uploaded (20)

Thoughts on MongoDB Analytics

Editor's Notes