Micro-Blogging for The Enterprise (MongoDB)

Editor's Notes

• #2: Enterprise is a closed ecosystem as compared to openness of Internet. \nThey have their own set of rules. \nYou must comply with internal technical standards, local regulations, enterprise architecture regulations etc.\nThis was no different and we started building the application using Enterprisy traditional relational database ==> which in this case was Oracle. \nThe db structure was simple with few tables but later the solution became convoluted by growing complex business rules. \nNot only this the performance of the app was degrading with growing volume of data.\nWe took a step back and translated the solution into elegance of document based db... yeah MongoDB\n\n
• #3: \n
• #4: \n
• #5: \n
• #6: \n
• #7: \n
• #8: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #9: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #10: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #11: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #12: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #13: I am going to focus on:\nWhat enterprise wants from micro-blogging tool. What are the major requirement\nThe kind of relational db structure we started with\nWith growing data volume the perf of the app started to degrade. We did some server side fine tuning but we were not solving the actual problem.\nCaching - this is the very first step we developer take, cache it and hide the problem under the covers :))\nHow MongoDB helped us and how we took the hybrid approach\n\n
• #14: \n
• #15: - Combination of traditional microblogging like twitter and social networking sites like facebook = enterprise micro-blogging\n- Best of the two worlds (thread like structure of activities)\n- Next lets have a look at some of the requirements we had to start with\n
• #16: - Combination of traditional microblogging like twitter and social networking sites like facebook = enterprise micro-blogging\n- Best of the two worlds (thread like structure of activities)\n- Next lets have a look at some of the requirements we had to start with\n
• #17: - Combination of traditional microblogging like twitter and social networking sites like facebook = enterprise micro-blogging\n- Best of the two worlds (thread like structure of activities)\n- Next lets have a look at some of the requirements we had to start with\n
• #18: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #19: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #20: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #21: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #22: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #23: - thread like structure of activities\n- Activities done by the people you follow flow to your timeline\n- Having to put filters was the tipping point to start the complexity in the code\n- Lets have a look at the typical Activity thread\n
• #24: - this is my timeline\n- Message from Joh Doe appears on my timeline as I follow him\n- Comment/Like are the ways to participate\n- I can filter threads based on role and office \n- Lets have a look at the DB structure we had to start with\n
• #25: - this is my timeline\n- Message from Joh Doe appears on my timeline as I follow him\n- Comment/Like are the ways to participate\n- I can filter threads based on role and office \n- Lets have a look at the DB structure we had to start with\n
• #26: - this is my timeline\n- Message from Joh Doe appears on my timeline as I follow him\n- Comment/Like are the ways to participate\n- I can filter threads based on role and office \n- Lets have a look at the DB structure we had to start with\n
• #27: - this is my timeline\n- Message from Joh Doe appears on my timeline as I follow him\n- Comment/Like are the ways to participate\n- I can filter threads based on role and office \n- Lets have a look at the DB structure we had to start with\n
• #28: - this is my timeline\n- Message from Joh Doe appears on my timeline as I follow him\n- Comment/Like are the ways to participate\n- I can filter threads based on role and office \n- Lets have a look at the DB structure we had to start with\n
• #29: \n
• #30: - the integration with Legacy database was one of the reason we started with relational Oracle db\n- lets quickly have at other set of requirements which started to make things more complex\n
• #31: - the integration with Legacy database was one of the reason we started with relational Oracle db\n- lets quickly have at other set of requirements which started to make things more complex\n
• #32: we treated hash tags as virtual person\nTo make matters even worse - we had to accommodate for \n\n
• #33: we treated hash tags as virtual person\nTo make matters even worse - we had to accommodate for \n\n
• #34: \n
• #35: \n
• #36: Two things to observe\n- Thread came to profile becoz I am following the hash tag\n- Martha (whom I follow commented)\n\nLets have a look at the impact on db structure coz of these set of requirements\n
• #37: Two things to observe\n- Thread came to profile becoz I am following the hash tag\n- Martha (whom I follow commented)\n\nLets have a look at the impact on db structure coz of these set of requirements\n
• #38: Two things to observe\n- Thread came to profile becoz I am following the hash tag\n- Martha (whom I follow commented)\n\nLets have a look at the impact on db structure coz of these set of requirements\n
• #39: Two things to observe\n- Thread came to profile becoz I am following the hash tag\n- Martha (whom I follow commented)\n\nLets have a look at the impact on db structure coz of these set of requirements\n
• #40: \n
• #41: \n
• #42: - two STIs\n- this is not the end\n- to build a visual threads with all the requirements we saw - we had to start writing hand crafted sql queries.\n- not the usual Rails way of using ActiveRecord\n- Let me show to you what they look like\n
• #43: - two STIs\n- this is not the end\n- to build a visual threads with all the requirements we saw - we had to start writing hand crafted sql queries.\n- not the usual Rails way of using ActiveRecord\n- Let me show to you what they look like\n
• #44: - two STIs\n- this is not the end\n- to build a visual threads with all the requirements we saw - we had to start writing hand crafted sql queries.\n- not the usual Rails way of using ActiveRecord\n- Let me show to you what they look like\n
• #45: - code became complex and unmaintainable\n
• #46: - code became complex and unmaintainable\n
• #47: - code became complex and unmaintainable\n
• #48: - code became complex and unmaintainable\n
• #49: - we did a quick benchmark of the sql queries\n
• #50: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #51: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #52: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #53: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #54: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #55: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #56: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #57: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #58: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #59: - How many user loads (10 user load test)\n- Environment of benchmark (mysql or oracle)\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\n- exponential increase in time\n\n- the first thing that struck us - lets cache baby !!\n- and the default option was Redis\n
• #60: \n
• #61: - but we soon realized that caching is not going to solve our problem, in fact it would have made matters more worse\n- we started looking at if document based db can help solve the problem at the same time scale with increasing complex business requirements\n
• #62: - but we soon realized that caching is not going to solve our problem, in fact it would have made matters more worse\n- we started looking at if document based db can help solve the problem at the same time scale with increasing complex business requirements\n
• #63: - but we soon realized that caching is not going to solve our problem, in fact it would have made matters more worse\n- we started looking at if document based db can help solve the problem at the same time scale with increasing complex business requirements\n
• #64: - but we soon realized that caching is not going to solve our problem, in fact it would have made matters more worse\n- we started looking at if document based db can help solve the problem at the same time scale with increasing complex business requirements\n
• #65: - which is true\n- each activity thread is like a small document\n
• #66: \n
• #67: \n
• #68: \n
• #69: - we are storing the user activity as relational structure\n- We convert the relational structure, run our business rules, convert them into JSON format\n- Give it to the View Layer to layout the result\n- Instead what we can do is store them as documents and cut short the whole deal of business rules etc…\n- Store them in the way business wants us to show…\n- we started to evaluating document based DB and again default was MongoDB\n- you might have this question\n
• #70: - we are storing the user activity as relational structure\n- We convert the relational structure, run our business rules, convert them into JSON format\n- Give it to the View Layer to layout the result\n- Instead what we can do is store them as documents and cut short the whole deal of business rules etc…\n- Store them in the way business wants us to show…\n- we started to evaluating document based DB and again default was MongoDB\n- you might have this question\n
• #71: - we are storing the user activity as relational structure\n- We convert the relational structure, run our business rules, convert them into JSON format\n- Give it to the View Layer to layout the result\n- Instead what we can do is store them as documents and cut short the whole deal of business rules etc…\n- Store them in the way business wants us to show…\n- we started to evaluating document based DB and again default was MongoDB\n- you might have this question\n
• #72: - I am pretty sure everyone sitting in this room is convinced why mongodb :)))\n- We took the hybrid approach as we had to deal with integrating Legacy Tables as well\n
• #73: - I am pretty sure everyone sitting in this room is convinced why mongodb :)))\n- We took the hybrid approach as we had to deal with integrating Legacy Tables as well\n
• #74: \n
• #75: - lets have a look at the simple arch diagram\n
• #76: We are still in the infant stages and there are so many things to learn from this community\n- the next obvious question how did we migrate the GBs of data from relational to document structure\n
• #77: - we already had the code to translate relational structure to json format (for view layer)\n- we used it to generate document structure for mongo\n- lets have a look at some benchmarks\n
• #78: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #79: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #80: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #81: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #82: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #83: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #84: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #85: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #86: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #87: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #88: SQL\n- 2m (200ms), 4m (600 ms), 8m (1100ms), 10m (1800ms), 12m (2500ms)\nMongo\n~ 280ms\n- so now our code is\n
• #89: \n
• #90: \n
• #91: \n
• #92: \n
• #93: \n
• #94: - not a full proof solution\n- but has worked for us \n
• #95: \n
• #96: \n
• #97: \n
• #98: \n
• #99: \n
• #100: \n
• #101: Few gotchas:\n- Do not forgot the schema design\n- We named the collection --> bloggers and later on changed to blogger\n\n