![GuiceyMongo?
• Supports
embedded
complex
objects,
lists,
maps,
sets,
enums
data Contact { data InstantMessenger {
data Address {
enum Application {
string street_1;
AIM,
string street_2;
ICQ,
string city;
Jabber,
string state;
MSN,
int zip_code;
Yahoo
}
}
[identity]
string screen_name;
string identity;
string alias;
IMApplication application;
string first_name; }
string last_name;
map<string, Address> address;
map<string, string> phone_number;
map<string, string> email_address;
map<string, InstantMessenger> instant_messenger;
set<string> tag;
blob picture;
}
8](https://image.slidesharecdn.com/mongonyc2011-110614022619-phpapp02/85/How-Signpost-uses-MongoDB-for-Tracking-and-Analytics-8-320.jpg){kind=tracking}
![Request
Tracking
‐
Queries
• Visitor
Keys
by
User
db.request.mapReduce(function()
{
emit(this.visitor_key,
{start:
this.timestamp,
end:
this.timestamp});
},
function(key,
values)
{
return
{start:
values[0].start,
end:
values[values.length
‐
1].end};
},
{
query:
{user_id:
12345},
sort:
{timestamp:
1}
});
• Requests
by
Visitor
Key
db.request.find({visitor_key:
‘a3896b987c98d798e791432fff332’}).sort({create_time:
‐1});
21](https://image.slidesharecdn.com/mongonyc2011-110614022619-phpapp02/85/How-Signpost-uses-MongoDB-for-Tracking-and-Analytics-21-320.jpg){kind=tracking}
![Event
Tracking
‐
Funnels
var
email_funnel
=
{
steps:
[{
type:
'event',
name:
'Unique
Emails
Sent',
output:
'length',
algorithm:
'distinct',
algorithm_data:
'event_properties.email_tracking_id',
query:
function()
{
var
query
=
{
event_name:
'email‐sent',
'event_properties.type':
'Daily
Deal',
create_time:
{}
};
query.create_time['$gte']
=
startDate;
query.create_time['$lt']
=
endDate;
return
query;
}
},
{
type:
'event',
name:
'Unique
Sessions
with
an
Email
Click',
output:
'length',
algorithm:
'distinct',
algorithm_data:
'session_key',
query:
{
event_name:
'email‐click',
'event_properties.email_tracking_id':
{$in:
'${data[0].data}'}
}
},
{
...
}]
};
execute_funnel(email_funnel);
26](https://image.slidesharecdn.com/mongonyc2011-110614022619-phpapp02/85/How-Signpost-uses-MongoDB-for-Tracking-and-Analytics-26-320.jpg){kind=tracking}
![Code?
function
sum_top_n_by_range(from,
to,
event,
property,
count)
{
var
r
=
db.event.rollup.mapReduce(function
()
{
emit(this.k[0].v,
this.c);
},
function
(k,
v)
{
var
s
=
0;
for
(var
i
in
v)
{
s
+=
v[i];
}
return
s;
},
{
query:
{e:
event,
s:
'day',
w:
{$gte:
from,
$lte:
to},
k:
{$size:
1},
'k.k':
property}
});
//
take
the
top
count
property_values
var
keys
=
r.find().sort({value:
‐1}).limit(count).map(function
(o)
{return
o._id;});
r.drop();
var
cursor
=
db.event.rollup.find({
e:
event,
s:
'hour',
w:
{$gte:from,
$lte:to},
k:
{$size:
1},
'k.k':
property,
'k.v':
{$in:
keys}
}).sort({w:
1});
var
result
=
{};
cursor.forEach(function(dataPoint)
{
//
organize
data
points
in
the
result
object
as
needed
by
your
output
});
return
result;
}
29](https://image.slidesharecdn.com/mongonyc2011-110614022619-phpapp02/85/How-Signpost-uses-MongoDB-for-Tracking-and-Analytics-29-320.jpg){kind=tracking}
![Event
Rollups
‐
Can
you
say
finance
background?
{
"_id"
:
ObjectId("4de54041c7d8327f56285563"),
"c"
:
1,
"e"
:
"deal‐click",
"k"
:
[{
"k"
:
"deal_at_location_id",
"v"
:
245931
}],
"s"
:
"month",
"w"
:
ISODate("2011‐05‐01T00:00:00Z")
}
• Allows
for
many
permutaDons
and
drill‐downs
• Easy
to
add
new
tracking
aSributes,
just
by
adding
key/value
pairs
• Easy
to
index
(queries
like
lightning)
32](https://image.slidesharecdn.com/mongonyc2011-110614022619-phpapp02/85/How-Signpost-uses-MongoDB-for-Tracking-and-Analytics-32-320.jpg){kind=tracking}
How Signpost uses MongoDB for Tracking and Analytics
- 2. Signpost ‐ Who are we? • AdWords meets Groupon ― LocaDon Based ― Merchant and User Contributed Deals ― Decentralized Sales PlaKorm 2
- 3. Who am I? • UDlity Knife Engineer • @maRnsler • hSp://github.com/maRnsler • hSp://www.maRnsler.com 3
- 4. Numbers • > 11M Tracked Requests • > 11M Tracked Events • 4 Developers • 2 Coffee runs / Day 4
- 5. What Makes it Work • Websites ― Java (Tomcat/Spring/MyBaDs), MySQL, Memcache • AnalyDcs ― Java (Tomcat/Cement/GuiceyMongo), MongoDB, Node.js 5
- 7. GuiceyMongo? • Generates Wrapper/Builder code from a simple DDL Person p = personCollection.findOne(); System.out(p.getName()); System.out(StringUtil.join(p.getAliasSet(), ", ")); if (p.hasPicture()) { Image picture = ImageIO.read(p.getPictureInputStream()); // do something with the picture } data Person { string name; set<string> alias; blob picture; } Person.Builder p = Person.newBuilder() .setName("Matt Insler") .addAlias("Matt") .addAlias("Guice & Mongo Guru") .setPictureBucket("pictures"); ImageIO.write(picture, format, p.getPictureOutputStream()); personCollection.save(p.build()); 7
- 8. GuiceyMongo? • Supports embedded complex objects, lists, maps, sets, enums data Contact { data InstantMessenger { data Address { enum Application { string street_1; AIM, string street_2; ICQ, string city; Jabber, string state; MSN, int zip_code; Yahoo } } [identity] string screen_name; string identity; string alias; IMApplication application; string first_name; } string last_name; map<string, Address> address; map<string, string> phone_number; map<string, string> email_address; map<string, InstantMessenger> instant_messenger; set<string> tag; blob picture; } 8
- 10. Cement? • MVC Framework wriSen for Guice/MongoDB • Efficient rouDng and built‐in logging and excepDon handling • Wrote it because I was bored and I could • Definitely not producDon‐ready whatsoever 10
- 12. We Track Everything • Request Tracking • Event Tracking • ExcepDon Tracking / Triage 12
- 13. We Track Everything • Request Tracking • Event Tracking • ExcepDon Tracking / Triage Every Fat-Fingered Form Submission 13
- 14. Request Tracking • Queued and wriSen from a low‐priority thread from Java ― WriSen to MySQL first ― WriSen to MongoDB with MySQL Primary Key for differenDaDon • Visitor Key ‐ Cookied browser/computer • Session Key ‐ Per User session • Stripe sessions with extra informaDon ― acquisiDon/referrer informaDon 14
- 15. AnalyDcs Architecture ‐ Request Tracking Flow Web MySQL MongoDB request 15
- 16. Request Tracking ‐ What we use this for • Spying on you • Customer Service • UX Analysis • Funnel Analysis • ExcepDon tracing and debugging 16
- 22. We Track Everything • Request Tracking • Event Tracking • ExcepDon Tracking / Triage 22
- 23. Event Tracking • Queued and wriSen from a low‐priority thread from Java ― WriSen to MySQL first ― WriSen to MongoDB with MySQL Primary Key for differenDaDon • Events are wriSen into a “raw” collecDon • Event Fixer process normalizes events into “fixed” collecDon ― Drops bot traffic ― Accounts for historical naming changes (user || userId || u ‐> user_id) ― Converts Strings to Numbers if necessary for later indexing • Event Indexer calculates and updates “rollup” collecDon 23
- 24. AnalyDcs Architecture ‐ Event Flow Web MySQL MongoDB event BI Event Fixer event.fixed Event Indexer event.rollup 24
- 25. Event Tracking ‐ What we use this for • Email Funnel Analysis • Purchase/Signup Funnels • Counts or Sums of event permutaDons over Dme ― We aggregate seconds, minutes, days, weeks, months, years • Top Count/Sum over a Dmeframe ― Such as top 5 deals by purchase in the past month 25
- 27. Event Tracking ‐ What we use this for • Email Funnel Analysis • Purchase/Signup Funnels • Counts or Sums of event permutaDons over Dme ― We aggregate seconds, minutes, days, weeks, months, years • Top Count/Sum over a Gmeframe ― Such as top 5 deals by purchase in the past month 27
- 28. Event Tracking ‐ What have we sold lots of? * Sorry, we can’t show you what these numbers actually are. 28
- 31. Events ‐ New Hotness { "_id" : ObjectId("4de53c85c7d8327f56284f3a"), "visitor_key" : "50f46d60e00efe0a17c42c7fffdc68bda530572d1f83972a83d6c61d684ba3ce", "event_name" : "deal‐click", "event_properties" : { "ca" : "dl", "deal_at_location_id" : 98303, "session_ca" : "dl", "ct" : "?", "cr" : "?", "node" : "web2", "bot" : "false", "user_id" : 163181, "guest" : "true", "session_cr" : "?", "session_ct" : "?" }, "session_key" : "1C9FA142A59465B745ECE2C1ACB15E62", "timestamp" : ISODate("2011‐05‐31T19:07:48.533Z"), "window" : { "minute" : ISODate("2011‐05‐31T19:07:00Z"), "hour" : ISODate("2011‐05‐31T19:00:00Z"), "day" : ISODate("2011‐05‐31T00:00:00Z"), "week" : ISODate("2011‐05‐30T00:00:00Z"), "month" : ISODate("2011‐05‐01T00:00:00Z"), "year" : ISODate("2011‐01‐01T00:00:00Z") } } 31
- 32. Event Rollups ‐ Can you say finance background? { "_id" : ObjectId("4de54041c7d8327f56285563"), "c" : 1, "e" : "deal‐click", "k" : [{ "k" : "deal_at_location_id", "v" : 245931 }], "s" : "month", "w" : ISODate("2011‐05‐01T00:00:00Z") } • Allows for many permutaDons and drill‐downs • Easy to add new tracking aSributes, just by adding key/value pairs • Easy to index (queries like lightning) 32
- 33. Tracking Events ‐ Problems • Rollups taking up too much space ― Need to move to document format from Kyle Banker’s “The MongoDB Gamut: Four ApplicaDon Designs” ― Tell business to track fewer permutaDons? (yea right) • Fixer skipping events ― Moved from {_id: {$gt: ObjectId(...)}} to {version: {$lt: 5}} • Fixer running slow ― Moved from findAndModify to find(...).limit(...) and then update in bulk ― {version: {$lt: 5}} is slower than {version: {$lte: 4}} !!! • KEEP INDEXES IN MEMORY!!! ― We had > 12G indexes on a 7.5G box 33
- 34. We Track Everything • Request Tracking • Event Tracking • ExcepGon Tracking / Triage 34
- 38. Make Users Happy! 38