Advanced Analytics & Statistics with MongoDB
4 likes3,241 views
The document discusses advanced analytics using MongoDB, highlighting the need for aggregate data insights and the introduction of a new statistical query language called Quirrel, designed for JSON data. It presents various features of Quirrel, including set-oriented queries, event handling, and filtering capabilities, as well as its composability and parallel processing advantages. Additionally, it contrasts Quirrel's expressive power with MongoDB's existing aggregation framework and suggests potential collaborations for enhancing data analysis functionalities.
![mongoDB
function map() {
emit(1, // Or put a GROUP BY key here
{sum: this.value, // the field you want stats for
min: this.value,
max: this.value,
count:1,
diff: 0, // M2,n: sum((val-mean)^2)
});
}
function reduce(key, values) {
var a = values[0]; // will reduce into here
for (var i=1/*!*/; i < values.length; i++){
var b = values[i]; // will merge 'b' into 'a'
// temp helpers
var delta = a.sum/a.count - b.sum/b.count; // a.mean - b.mean
var weight = (a.count * b.count)/(a.count + b.count);
// do the reducing
a.diff += b.diff + delta*delta*weight;
a.sum += b.sum;
a.count += b.count;
a.min = Math.min(a.min, b.min);
a.max = Math.max(a.max, b.max);
}
return a;
}
function finalize(key, value){
value.avg = value.sum / value.count;
value.variance = value.diff / value.count;
value.stddev = Math.sqrt(value.variance);
return value;
}](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-5-320.jpg)
![mongoDB
(really) basic queries
quirrel> 1
[1]
quirrel> true
[true]
quirrel> {userId: 1239823, name: “John Doe”}
[{userId: 1239823, name: “John Doe”}]
quirrel>1 + 2
[3]
quirrel> sqrt(16) * 4 - 1 / 3
[5]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-13-320.jpg)
![mongoDB
loading data
quirrel> //payments
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]
quirrel> load(“/payments”)
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-14-320.jpg)
![mongoDB
variables
quirrel> payments := //payments
| payments
[{"amount":5,"date":1329741127233,"recipients":
["research","marketing"]}, ...]
quirrel> five := 5
| five * 2
[10]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-15-320.jpg)
![mongoDB
filtered descent
quirrel> //users.userId
[9823461231, 916727123, 23987183, ...]
quirrel> //payments.recipients[0]
["engineering","operations","research", ...]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-16-320.jpg)
![mongoDB
identity matching
quirrel> orders := //orders
| orders.subTotal +
| orders.subTotal *
| orders.taxRate +
| orders.shipping + orders.handling
[153.54805, 152.7618, 80.38365, ...]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-19-320.jpg)
![mongoDB
values
quirrel> payments.amount * 0.10
[6.1, 27.842, 29.084, 50, 0.5, 16.955, ...]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-20-320.jpg)
![mongoDB
filtering
quirrel> users := //users
| segment := users.age > 19 &
| users.age < 53 & users.income > 60000
| count(users where segment)
[15]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-21-320.jpg)
![mongoDB
user functions
quirrel> pageViews := //pageViews
|
| statsForUser('userId) :=
| {userId: 'userId,
| meanPageView: mean(pageViews.duration
| where pageViews.userId = 'userId)}
|
| statsForUser
[{"userId":12353,"meanPageView":100.66666666666667},{"userId":
12359,"meanPageView":83}, ...]](https://image.slidesharecdn.com/mongoandquirreldeck-120504164406-phpapp02/85/Advanced-Analytics-Statistics-with-MongoDB-23-320.jpg)
Advanced Analytics & Statistics with MongoDB
- 1. mongoDB advanced analytics and statistics with mongodb John A. De Goes @jdegoes http://precog.io 04/30/2012
- 2. mongoDB what do you want from your data?
- 3. mongoDB I want to get and I want aggregates I want deep insight put data MongoDB MongoDB Query Aggregation ??? Language Framework SQL data storage data intelligence
- 4. mongoDB I want to get and I want aggregates I want deep insight put data MongoDB MongoDB Map Query Aggregation Reduce Language Framework SQL data storage data intelligence
- 5. mongoDB function map() { emit(1, // Or put a GROUP BY key here {sum: this.value, // the field you want stats for min: this.value, max: this.value, count:1, diff: 0, // M2,n: sum((val-mean)^2) }); } function reduce(key, values) { var a = values[0]; // will reduce into here for (var i=1/*!*/; i < values.length; i++){ var b = values[i]; // will merge 'b' into 'a' // temp helpers var delta = a.sum/a.count - b.sum/b.count; // a.mean - b.mean var weight = (a.count * b.count)/(a.count + b.count); // do the reducing a.diff += b.diff + delta*delta*weight; a.sum += b.sum; a.count += b.count; a.min = Math.min(a.min, b.min); a.max = Math.max(a.max, b.max); } return a; } function finalize(key, value){ value.avg = value.sum / value.count; value.variance = value.diff / value.count; value.stddev = Math.sqrt(value.variance); return value; }
- 6. mongoDB what if there were another way?
- 7. mongoDB introducing • Statistical query language for JSON data • Purely declarative • Implicitly parallel • Inherently composable
- 8. mongoDB a taste of quirrel pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) avg := mean(pageViews.duration) lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId
- 9. mongoDB a taste of quirrel pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) Users who spend an unusually avg := mean(pageViews.duration) long time looking at a page! lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId
- 10. mongoDB quirrel in 10 minutes
- 11. mongoDB set-oriented in Quirrel everything is a set of events
- 12. mongoDB event an event is a JSON value paired with an identity
- 13. mongoDB (really) basic queries quirrel> 1 [1] quirrel> true [true] quirrel> {userId: 1239823, name: “John Doe”} [{userId: 1239823, name: “John Doe”}] quirrel>1 + 2 [3] quirrel> sqrt(16) * 4 - 1 / 3 [5]
- 14. mongoDB loading data quirrel> //payments [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...] quirrel> load(“/payments”) [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...]
- 15. mongoDB variables quirrel> payments := //payments | payments [{"amount":5,"date":1329741127233,"recipients": ["research","marketing"]}, ...] quirrel> five := 5 | five * 2 [10]
- 16. mongoDB filtered descent quirrel> //users.userId [9823461231, 916727123, 23987183, ...] quirrel> //payments.recipients[0] ["engineering","operations","research", ...]
- 17. mongoDB reductions quirrel> count(//users) 24185132 quirrel> mean(//payments.amount) 87.39 quirrel> sum(//payments.amount) 921541.29 quirrel> stdDev(//payments.amount) 31.84
- 18. mongoDB identity matching a*b a e1 ? b e2 e8 e3 e9 e4 * e10 e5 e11 e6 e12 ? e7
- 19. mongoDB identity matching quirrel> orders := //orders | orders.subTotal + | orders.subTotal * | orders.taxRate + | orders.shipping + orders.handling [153.54805, 152.7618, 80.38365, ...]
- 20. mongoDB values quirrel> payments.amount * 0.10 [6.1, 27.842, 29.084, 50, 0.5, 16.955, ...]
- 21. mongoDB filtering quirrel> users := //users | segment := users.age > 19 & | users.age < 53 & users.income > 60000 | count(users where segment) [15]
- 22. mongoDB chaining pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) avg := mean(pageViews.duration) lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId
- 23. mongoDB user functions quirrel> pageViews := //pageViews | | statsForUser('userId) := | {userId: 'userId, | meanPageView: mean(pageViews.duration | where pageViews.userId = 'userId)} | | statsForUser [{"userId":12353,"meanPageView":100.66666666666667},{"userId": 12359,"meanPageView":83}, ...]
- 24. mongoDB lots more! • Cross-joins • Self-joins • Augmentation • Power-packed standard library
- 25. mongoDB quirrel -> mongodb • Quirrel is extremely expressive • Aggregation framework insufficient • Working with 10gen on new primitives • Backup plan: AF + MapReduce
- 26. mongoDB quirrel -> mongodb pageViews := //pageViews bound := 1.5 * stdDev(pageViews.duration) one-pass avg := mean(pageViews.duration) map/reduce lengthyPageViews := pageViews where pageViews.duration > (avg + bound) lengthyPageViews.userId one-pass mongo filter
- 27. mongoDB qa John A. De Goes @jdegoes http://precog.io 04/30/2012