![Collections
❖ Customers
❖ Payments [array]
❖ Orders
❖ Orderdetails
❖ Employees
❖ Products
❖ Productlines
❖ Offices](https://image.slidesharecdn.com/fromsqltomongodbpresentation-170913135205/85/From-SQL-to-MongoDB-10-320.jpg)
![Collections
❖ Customers
❖ Payments [array]
❖ Orders
❖ Orderdetails [array]
❖ Products [document]
❖ Productlines [document]
❖ Offices
❖ Products
❖ Productlines](https://image.slidesharecdn.com/fromsqltomongodbpresentation-170913135205/85/From-SQL-to-MongoDB-12-320.jpg)
![Example - SELECT
❖ SELECT * FROM orders, orderdetails, products WHERE
…. [1000]
❖ SELECT * FROM offices, employees WHERE … [100]
❖ SELECT * FROM productlines, products WHERE … [2000]
❖ SELECT * FROM products WHERE … [4000]
❖ SELECT * FROM employees, customers WHERE … [200]
❖ SELECT * FROM customers, orders WHERE … [200]](https://image.slidesharecdn.com/fromsqltomongodbpresentation-170913135205/85/From-SQL-to-MongoDB-16-320.jpg)
![❖ Customer[1:n] -> Payment relationship
❖ A payment created at T=5, T=50
❖ Customer[1:n] -> Orders
❖ An order created at T=0, T=15, T=20, T=45
❖ Unbound Relationships
Customer Life Span Example](https://image.slidesharecdn.com/fromsqltomongodbpresentation-170913135205/85/From-SQL-to-MongoDB-25-320.jpg)
![Collections
❖ Customers
❖ Payments [array]
❖ Orders
❖ Orderdetails [array]
❖ Products [document]
❖ Productlines [document]
❖ Offices
❖ Products
❖ Productlines](https://image.slidesharecdn.com/fromsqltomongodbpresentation-170913135205/85/From-SQL-to-MongoDB-36-320.jpg)
From SQL to MongoDB
- 1. Christian Amor Kvalheim (MongoDB Staff Engineer) From SQL to MongoDB How to get from A to B in a reasonably ordered fashion
- 2. Whats Up ❖ The Challenge ❖ Explicit Schema ❖ Implicit Schema ❖ Rules of Thumb ❖ Summary
- 3. The Challenge Take an existing SQL Schema and pick an Appropriate MongoDB Schema
- 4. Our Example SQL Schema
- 5. Explicit Schema ❖ Table structure definition ❖ Primary Key definition ❖ Foreign Key relationships ❖ 1:n ❖ 1:1 ❖ n:m
- 6. Implicit Schema ❖ The SQL Schema as expressed by the following operations and their associated metadata ❖ Insert operation ❖ Update operations ❖ Select Operations ❖ Join relationships
- 7. With Explicit Schema Only
- 9. No duplication of Data 1 n 1 n n 1 1 n 1 n 1 n 1 n 1 n
- 10. Collections ❖ Customers ❖ Payments [array] ❖ Orders ❖ Orderdetails ❖ Employees ❖ Products ❖ Productlines ❖ Offices
- 11. What if we allow duplication ? 1 n 1 n n 1 1 n 1 n 1 n 1 n 1 n
- 12. Collections ❖ Customers ❖ Payments [array] ❖ Orders ❖ Orderdetails [array] ❖ Products [document] ❖ Productlines [document] ❖ Offices ❖ Products ❖ Productlines
- 13. Important Notes ❖ Foreign Key Relationship in most cases are not representative of application level queries ❖ Cannot discover the degree of mutability looking at the SQL in isolation ❖ Cannot know how the average sizes of n in the 1:n relationships
- 14. Implicit Schema
- 15. Implicit Schema ❖ The Implicit Schema represents the SQL operations executed against the relational schema (Application Schema) ❖ Can vary hugely from the foreign key relationships ❖ Expresses read/vs write ratios for tables ❖ Can be used to deduct entity mutability ❖ Can be used to estimate n in the 1:n relationships
- 16. Example - SELECT ❖ SELECT * FROM orders, orderdetails, products WHERE …. [1000] ❖ SELECT * FROM offices, employees WHERE … [100] ❖ SELECT * FROM productlines, products WHERE … [2000] ❖ SELECT * FROM products WHERE … [4000] ❖ SELECT * FROM employees, customers WHERE … [200] ❖ SELECT * FROM customers, orders WHERE … [200]
- 17. What We Can Learn ❖ The frequency of the SQL operations ❖ The Application Schema relationships studying the join relationships. ❖ If the logs include the number of rows returned we can make estimates for the size of n in the 1:n relationships ❖ We can also calculate the rate of growth of the n over time
- 18. 1 ~5 (+1 every 100 min) 1 ~12 1 1 1 ~10001 ~15 1 ~10 1 ~20 1 ~6
- 19. Example - INSERT/UPDATE ❖ INSERT (…) VALUES (…) INTO orders ❖ INSERT (…) VALUES (…) INTO order details ❖ UPDATE … orders WHERE orderNumber = 1
- 20. Data Islands
- 21. ❖ Single Item Mutability Rate (SIMR) ❖ How much an entity mutates in a given time period ❖ A low mutation rate ❖ Entity reaches a stable state and is a good candidate for rolling up into a single document ❖ Duplication of data is ok as the document is a snapshot in time ❖ A high mutation rate ❖ Entity does not reach a stable state and keeps mutating and might not be a good candidate for rollup Single Item Mutability Rate
- 22. ❖ Order life span example ❖ An order gets created at T=0 ❖ 10 order details are created at T+1 ❖ Order is filled and order record updated T+10 ❖ Order is shipped and order record updated T+15 ❖ Past T+15 there are no more mutations Order Life Span Example
- 23. Order Life Span Example T T = 0 Order Created T = 1 Added 10 Order Details T = 10 Order Fulfilled T = 15 Order Shipped
- 25. ❖ Customer[1:n] -> Payment relationship ❖ A payment created at T=5, T=50 ❖ Customer[1:n] -> Orders ❖ An order created at T=0, T=15, T=20, T=45 ❖ Unbound Relationships Customer Life Span Example
- 26. Customer Life Span Example T T = 0 Order Created T = 1 T = 5 Payment Created T = 15 Order Created Order Created T = 20 Order Created
- 27. 1 1 ~12 1 1 1 ~10001 ~15 1 ~10 1 ~20 1 ~6 ~5 (+1 every 5 days)
- 28. ❖ The recursive relationship for Employees makes it unsuitable for rolling up ❖ The same recursive relationship also affects the offices- >employees relationship ❖ The ProductLines -> Products relationship are big and possibly unbound And The Rest ?
- 29. Rules of Thumb
- 30. 1. SQL Schema + Foreign Key Relationships ❖ Only have the Explicit Relationships and Table definitions 2. SQL Operations Logs (mysql general log) ❖ Contains only SQL operations (no result set size) 3. Full SQL Operations Logs (mysql slow log) ❖ Contains SQL operations (result set size, latencies) Levels Of Information
- 31. 1. Use Selects with Joins to draw the new relationship 2. Establish the average n join relationship 3. Establish the mutation rate of over time ❖ Does the relationship go static ? ❖ Are the relationships unbound ? (growing n) Analysis Steps
- 32. 1. Roll up relationships 1. If entity relationship reaches a static state 2. If the rate of growth of n is slow enough for the relationship to be static (analyst discretion) 2. Don’t rollup relationships 1. If the rate of mutability is high 2. If the average size of n is huge 3. If the mutation rate of the entity is large 4. If an entity has a recursive relationship Algorithms
- 33. Applying It
- 34. 1 1 ~12 1 1 1 ~10001 ~15 1 ~10 1 ~20 1 ~6 ~5 (+1 every 5 days) Collapsing, Duplicating Products and Productlines
- 35. 1 1 ~12 1 1 1 ~10001 ~15 1 ~10 1 ~20 1 ~6 ~5 (+1 every 5 days) Collapsing, Duplicating Products and Productlines Collapsing Payments into Customers as never queried separately
- 36. Collections ❖ Customers ❖ Payments [array] ❖ Orders ❖ Orderdetails [array] ❖ Products [document] ❖ Productlines [document] ❖ Offices ❖ Products ❖ Productlines
- 37. What’s Cooking
- 38. 1. We are working on building tooling to help 1. Analyze your relational schema 2. Propose schema recommendations 3. Load and transform your data 2. Push the whole subject of schema transformation forward doing something never done before Tooling
- 39. 1. Are Operation Latencies important for recommending a Schema ? 2. Can one quantify a schema recommendation (is recommendation A better than B and if, then why ?) 3. Can Machine Learning produce better recommendations ? 4. … etc etc Tons of Open Questions
- 40. Are you ready to build a new team, to build a brand new product, and to create a whole new category of products for the most popular NoSQL database? MongoDB, the leader in NoSQL databases is building a new team in Dublin. This team will develop products that help our customer adopt our technology by analyzing their legacy relational systems. We need someone who is going to participate in the research, partner with our staff engineers who are prototyping solutions, write production ready code, and build a team. This person will report to the Director of Integrations at MongoDB. Come Work With Us http://grnh.se/ge1rfp1