Migration from SQL to MongoDB - A Case Study at TheKnot.com

| 1
© 2014 XO GROUP INC. ALL RIGHTS RESERVED.

| 2
XO Group Inc.
Membership and Community Team
Alexander Copquin - Senior Software Engineer
Vladimir Carballo - Senior Software Engineer

| 3
Favorites API Re-platforming
…a case study

| 4
Favorites API Re-platforming
• Architectures SQL .NET / Ruby Mongo
• Reasons for migration
• Schema design
• RoR model design and implementation
• Migration strategies and systems
• Lessons learned

| 5
Our Favorites Feature

| 6
Favorites API
• Add / Edit / Delete Object.
• Manage Boards
• Get counts & stats
• RESTful API
• Rails
• JavaScript
• Ios
• Android
Features

| 7
• + 100,000,000 “favorited” objects
• + 760,000 boards
• Avg. 55,000 new objects per day
Stats

| 8
Legacy Architecture

| 9
• Database 55 GB and growing…
• Avg 45 rpm on peak times
• Avg 80 msec response POST
• Avg 460 msec response GET
Legacy Benchmarks

| 10
• Db Reaching max. capacity for setup
• Scalability problems
• Hard to modify schema
• Bad response times
• Very complex caching layer
• Out of line with company’s strategy
Maxed

| 11
New Architecture

| 12
• Easy to scale
• Flexible schema
• Fast Response
• No Cache Layer
• Fast Iteration / Deploy
• TDD first and foremost
• At a glance monitoring of all layers
Scalable

| 13
Implementation

| 14
What we persisted in the legacy schema
• UserId (primary key)
• UniqueId
• Url (unique per user)
• ImageUrl
• Name
• Description
• ObjectId (unique per application
adding favorites)
• Category
• Timestamps
• Other

| 15
Favorites DB Legacy Schema

| 16
select top 10 UserFavoriteId, Name, Description, Url, ImageUrl
from userFavorites
where userId = '5174181997807393'
Sample queries

| 17
select top 5 grp.groupId, grp.Name as GroupName,
fav.userFavoriteId, fav.name,
fav.Description, fav.Url, fav.ImageUrl
from userFavoritesGroups grp
inner join userFavoritesGroupsItems grpItm
on grp.GroupId = grpItm.GroupId
inner join userFavorites fav
on grpItm.userFavoriteId = fav.userFavoriteId
where grp.userId = '5174181997807393'
Sample queries

| 18
Towards a new Schema and Persistance Layer
• Start with a clean slate
• Break with the past
• Persist only relevant minimum data points
• Think and rethink relationships
• High Performance
• Flexible
• Prototype different scenarios

| 19
First attempt

| 20
UserFavorites

| 21
• Document contains embedded documents which are
required to be accessed on its own
• Documents would grow without bound
• Most queries would be slow
• Indexes would be very expensive
• Tries too hard to imitate legacy
Cons

| 22
Second attempt

| 23

| 24
Board document with one recent favorite

| 25
Board document with more recent favorites

| 26

| 27
Favorite document located on different boards

| 28
• Document structure matches the data required on the view
• A Board document includes the 4 most recent favorites.
• A Favorite document includes the list of boards it was
added to
• Faster queries.
• More control on the size of each document
• Better implementation of UX intent
Pros

| 29
Sample queries
db.favorites.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903'})
.limit(1)
db.favorites.find({'boards': '7557acf8-b7b1-4eab-a64d-57449034cfc6'})
.limit(1)
db.favorites.find({'application': 'marketplace'})
.limit(1)
db.boards.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903'})
.limit(1)
db.boards.find({'member': 'e1606ed5-4ac8-48b4-aee6-bc4203937903',
'default_board': true})
db.boards.find({'name' : 'Simple Reception Decor'}).limit(1)

| 30
● Rails web application framework
● We speak RoR and JS
● mongoDB as a data repository (we love NoSQL)
● Two collections, one for Boards and one for
Favorites
● No joins, no foreign keys
● Referential integrity is handled in a different fashion.
● MongoId Gem (Pros & Cons)
Some implementation details

| 31
Favorites re-platform

| 32
Board
class

| 33

| 34
Favorite
class

| 35

| 36
Scaling reads with replica sets

| 37
Scaling reads with sharding

| 38
Migration

| 39
Clients switchover
New
API
Legacy API
Client
Client
Client
Client
ONE WAY

| 40
Migration Timeline
new API.
Continuous
Migr.
Implement
Monitors
Turn on
Continuous
Data
Catch-up
Plug ClientsBulk Migr.
Development
Bulk Migr.
Migration

| 41
Bulk Migration
ETL

| 42
Bulk Migration
FavoritesUserFavorites
SQL Tables Mongo Collections
BoardsUserFavoritesGroups
UserFavoritesGroupsItems

| 43
Favorites Job

| 44
Pentaho Steps

| 45
Auto-increment Id vs. UUID
UserFavoriteId
GroupId
Favorites UUID
Groups UUID
Continuous
Migration

| 46
Get UUID from the Get Go
• Add a column to legacy Db (+ 100M recs!!) with new
Mongo UUID
• Then migrations will take care of inserting into new
documents
SQL
has all new
ids
xxxxx-xxxx-xxxx
xxxxx-xxxx-xxxx
xxxxx-xxxx-xxxx
Mongo
Ids are
inserted
Migration
Systems

| 47
Add UUID Columns in SQL
100 M recs!!!Alter table add UUID uniqueidentifier
New Favs TempTable
with UUID
SELECT *, uuid = NEWID()
INTO NewUserFavorites
FROM UserFavorites
Add
Indexes
Rename & drop

| 48
• SQL needed some sanitation
• SQL prep scripts approx. 4 hs
• Pentaho ETL on local Workstation: 8hs
• Restore into production Mongo Cluster: 4hs
Facts

| 49
We’ve got data!!

| 50
Continuous Migration Architecture
Clients
Legacy API New API
SQS Queue Messenger
ONE WAY SYNC…

| 51
Continuous Migration
Favorites Legacy Messenger
• Ruby
• Consumer of an SQS queue coming from Legacy
that generates 1 message per operation
• Issues API call to new app per each operation
• Runs as a worker in the background
Legacy
API
SQS
Legacy
Messenge
r
New
API
Mongo

| 52
SQS Queue is not a FIFO Friend
Sent by Legacy
1
2
3
4
5
6
7
5
3
1
2
4
6
7
Consumed by Messenger

| 53
• Queue is not FIFO
• Objects don’t exist
• Queue bloats fast
• Can get like not-real-time
• Data is different
Challenges

| 54
• Verify if entity exists (API call),
otherwise, throw back in queue
• Set message expiration
• Sanitize data
• Get multiple workers to achieve
near real-time syncing.
Solutions

| 55
• Favor a simple document structure
• Try different schema paradigms
• Bypass native objectId generation in favor of UUID
• Break with the past
• Queues can be deceiving
• Gems can simplify application layer impl.
• Manage ref. integrity in app. layer
• No cache required Take away

| 56
• Avg 85 rpm on peak times
• Avg 58 msec response POST
• Avg 18 msec response GET
New Benchmarks

| 57
New vs. Legacy
• Overall Performance Increase
• 18 ms vs. 460 ms for GET
• 58 ms vs. 80 ms for POST
• Easy Schema Changes
• Scalable
• Simpler architecture
• No Cache layer
• Fast Code iteration, testing and deployment
• In-line with company’s technology strategy
Good

| 58
Acknowledgments
• Dmitri Nesterenko
• Jason Sherman
• Nelly Santoso
• Phillip Chiu
• Sean Lipkin
• George Taveras
• Alison Fay
• Diana Taykhman
• Rajendra Prashad
• Josh Keys
• Lewis DiFelice

| 59
contact, questions, inquiries?
memcomtech@xogrp.com

Migration from SQL to MongoDB - A Case Study at TheKnot.com

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Migration from SQL to MongoDB - A Case Study at TheKnot.com (20)

More from MongoDB (20)

Recently uploaded (20)

Migration from SQL to MongoDB - A Case Study at TheKnot.com