mongoDB at Visibiz

Editor's Notes

• #2: thanks for joining us today, i am mike brocious, tech lead here at visibiz, and \ni’m gonna take you through a brief discussion of why we chose mongodb and how it fits into our application\n\nTurn off all message notifications\n- tweet deck\n- gmail\n- skype\n\n
• #3: why we chose mongo as primary datastore\n\nNot a sales pitch - Can’t tell you whether it’s right for you\n\nHow we’ve fit it into our architecture\n\n
• #4: startup\nbeen around about a year\n\nour application is in the social CRM market space - take traditional CRM features, throw in social networks, email and some intelligence...we help you understand your network of contacts, and prospects to help you during the sales cycle\n\n\n
• #5: a little more context about the application we’re developing it in the Grails framework from SpringSource\nso grails application hosted inside of tomcat, front-ended by apache with mongo, recently added elastic search as search engine\nall hosted in the amazon ec2 cloud\n
• #6: a lot factors went into the decision, but there are two worth noting here\n-we want our customers to be able to extend the core domain objects\n we also want them to be able to define their own domain objects and plug them into our application\n-the other point is scalability....important to us that our datastore be prepared to scale up and scale out as necessary\n\n
• #7: so lets talk a little more about extensibility\nwe provide core objects with a set of attributes\nto help customers to fit our application into their own workflow, we want customers to be able to extend those objects with their own attributes\nso what this means is that a person object ........\n
• #8: C#1 is interested in tracking the employment history of people that are in the network\nC#2 need to know are they male or female and what they do in their spare time, what are their interests, what are their hobbies\n\nthose attributes add meaning for those customers based on their workflow, their marketspace, their targets\nobviously a trivial example, but it’s meant to give you a feeling for what we want to want to be able to offer in our application\n
• #9: no real business logic, just the ability to store, associate and retrieve their own objects\n\n
• #10: Second concern that i mentioned earlier\nWanted to make sure that the datastore that we chose was designed with scalability in mind\nI mentioned that our application is dealing with social networks. as most of you are probably a member of one or more networks, be it twitter, facebook, linkedin, there’s a ton of data out there related to social networks. but even beyond the social networks, we’re going to be getting into email, blogs, articles, and whatever else is on the web that can provide information about your network....so we’re going to be housing a lot of data...\nwe want scaling to be easy, shouldn’t be rocket science....want to focus on developing the application, and not administering/supporting db clusters\n
• #11: So...given that criteria a backdrop, we looked at our options: relational, couple of NoSQL / schema-free datastores\nnot exhaustive evaluation, but enough to feel comfortable....obviously we chose mongo, or we wouldn’t be having this conversation\n\nmakes it that much easier to support extensible objects...lots of other challenges, but at least the database is not one of them\nas a developer, you deal with your domain data on a daily basis....it’s so nice to see the data presented in a way that easy for humans to understand\n
• #12: So...given that criteria a backdrop, we looked at our options: relational, couple of NoSQL / schema-free datastores\nnot exhaustive evaluation, but enough to feel comfortable....obviously we chose mongo, or we wouldn’t be having this conversation\n\nmakes it that much easier to support extensible objects...lots of other challenges, but at least the database is not one of them\nas a developer, you deal with your domain data on a daily basis....it’s so nice to see the data presented in a way that easy for humans to understand\n
• #13: \n
• #14: Had a design roughed out in relational model\n- team member had implemented something similar in previous life\n\n\n
• #15: \n
• #16: of course the name ‘mongo’ comes from humongous so that’s a pretty good indication that scalability is built-in\n\nfor us, JSON is new XML, it’s everywhere\n\nso that’s one thing that’s good about JSON, the other thing is....\n\n(drill into this a little)\n\n
• #17: of course the name ‘mongo’ comes from humongous so that’s a pretty good indication that scalability is built-in\n\nfor us, JSON is new XML, it’s everywhere\n\nso that’s one thing that’s good about JSON, the other thing is....\n\n(drill into this a little)\n\n
• #18: Take a look at the major components of our application\n\nHad already chosen Grails as app framework\n\nwhat this really means....\n
• #19: +Can move really fast, fewer moving parts to develop/maintain\n+Good to get application going, plan on refactoring \n\n
• #20: 10gen JIRA list is active\n\nClear from mongo forums, etc. that mongo is very active product.\n- not quite as sure with some of the other projects\n\n- not alone\n\n
• #21: of course, nothing is perfect, life is a bunch of trade-offs\n\nyou don’t have transactions or really complex queries....JOINs, nested SELECTs\n\nlack of transactions was the thing I was most nervous about. have to go into with your eyes wide open, have to understand how to minimize the risk, and reduce the impact.\n
• #22: \n
• #23: \n
• #24: the name 'things' represents that this collection will hold different object types\nhaving all Things in one collection allows us to query across all Things\ngood use of Mongo's 'schema free' document store\n show example thing (JSON)\n\n
• #25: \n
• #26: \n
• #27: \n
• #28: first cut: id of thing on left, thing on right and label\nwe were still thinking in relational terms\n\nwill be reading relationships much more often than creating new ones, so slow queries is definitely a bad thing\n\n
• #29: \n
• #30: \n
• #31: \n
• #32: \n
• #33: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #34: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #35: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #36: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #37: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #38: hybrid solution - gives us best of both worlds\n- insertion integrity\n- query performance\ndownside:\nnow inserting/updating a rel is a two-step process: update “rels” collection, then push the changes to the related things\nif that second update fails, we can always rebuild those relationships from the rels collection.\n\n
• #39: the last major thing i want to talk about came about because we’re supporting a RESTful interface and mongo doesn’t enforce datatypes.\n\nmongo is not going to stop you from storing any kind of data in any field - it’s ‘schema-free’.\nenter JSON schema - provides a way to describe an object using JSON syntax\n\n
• #40: analogous to DTD or XML schema\n
• #41: \n
• #42: one last point....the ease with which you can create collections, and the fact that you can store different looking objects in the same collection opens up possibilities\n\n\n
• #43: \n
• #44: \n