Realtime Analytics with MongoDB Counters (mongonyc 2012)
Download as PPTX, PDF6 likes2,760 views
This document discusses pre-aggregating data with counters in MongoDB to enable dashboard-style reports. It involves defining metrics and aggregations upfront, and using the $inc operator to increment values in aggregation collections during write operations. This allows generating real-time aggregated data without requiring map-reduce or the aggregation framework, providing better performance for dashboard queries.
![Stats Collections
stats_[hourly/daily/monthly].actors
stats_[hourly/daily/monthly].projects
stats_[hourly/daily/monthly].langs
stats_[hourly/daily/monthly].types](https://image.slidesharecdn.com/mongonyc2012-realtimeanalytics-120601141250-phpapp01/85/Realtime-Analytics-with-MongoDB-Counters-mongonyc-2012-13-320.jpg)
Realtime Analytics with MongoDB Counters (mongonyc 2012)
- 1. Pre-aggregation with counters © Copyright 2010 10gen Inc.
- 2. Goals • Dashboard style reports • (Known) Reports • Real-time numbers
- 3. Framework • Know your metrics/counter • Prepared reports • Calculate during write • Fast queries • Always up to date data • Record time-series collections
- 4. Rationale • Documents are updated in-place* • $inc update operator • Working set is small • Aggregations are much smaller*
- 5. Dashboard Projects Lines Events 6497 5543 3401 3543 2314 2342 921 1234 16 27 42 45 JavaScript Java Ruby Python Monday Tuesday Thursday Friday
- 7. Roads not traveled • Map/Reduce • Reprocess raw data • Now possible to do partial reduce • Aggregation Framework (aggregate in 2.2) • Also reprocess data on operation (initial release) • Optimizations to come • More costly during reads
- 8. Not Appropriate For • Ad-hoc aggregations (unknown metrics) • One-off reports • Possibly complex calculations
- 9. Processing • Event received • Split into many updates w/$inc • Aggregate • Input Field(s) • Time periods (hourly, daily, monthly) • Defined Metrics
- 10. Example Data: github > db.events.findOne() { "repository" : { "url" : "https://github.com/vidageek/games", ... "open_issues" : 25, "watchers" : 6, "pushed_at" : "2012/03/10 08:34:00 -0800", "language" : "Java" }, "actor_attributes" : {...}, "created_at" : "2012/03/11 15:20:24 -0700", "public" : true, "actor" : "juliano", "payload" : {...}, "url" : "https://github.com/...", "type" : "CommitCommentEvent” }
- 11. Define Metrics • “actor” • “repository.name” • “repository.language” • “type” PushEvent, IssuesEvent, WatchEvent, GistEvent • “payload.ref” efs/heads/improved_history, refs/heads/master, refs /heads/signs
- 12. Aggregations TimePeriod, type # TimePeriod, author # TimePeriod, project #
- 14. Stats > db.stats_hourly.types.find({"_id.type":"GistEvent"}) { "_id" : { "p" : ISODate("2012-05-21T00:00:00Z"), "type" : "GistEvent” }, "hour" : { "2" : { "count" : 65 }, "3" : { "count" : 2 }, "7" : { ”count" : 130}, "8" : { "count" : 5 } }, "total" : { ”count" : 202 } }
- 15. Updates Increment Query: { ”p" : Date(…), "actor" : "neoplastic"}} Update: { "$inc" : { "h.21.c" : 1 , "t.c" : 1}} Upsert : true
- 16. Query/Graphing • Select by grouping (by date, by type/value) • Documents hold many data points
- 17. The Whys • Writing more data up front, helps with reads • Multiple data points per document • Documents hold many timed points • Good for graphs by time, or types • Nested for improved performance
- 18. Thanks for coming… ne questions
- 19. drivers at mongodb.org mongodb.org Supported Community Supported C REST node.js C# ActionScript3 Objective C C++ C# and .NET PHP Erlang Clojure PowerShell Haskell ColdFusion Blog post Java Delphi Python Javascript Erlang Ruby Perl F# Scala PHP Go: gomongo Scheme (PLT) Python Groovy Smalltalk: Dolphin Ruby Haskell Smalltalk Javascript Lua © Copyright 2010 10gen Inc.
- 20. download at mongodb.org conferences, appearances, and meetups http://www.10gen.com/events Facebook | Twitter | LinkedIn http://bit.ly/mongofb @mongodb http://linkd.in/joinmongo support, training, and this talk brought to you by © Copyright 2010 10gen Inc.