Lightning Talk: What You Need to Know Before You Shard in 20 Minutes
Download as PPTX, PDF1 like1,051 views
The document discusses sharding in MongoDB, providing insights on why, who, when, and where to shard a database. It emphasizes the importance of scaling out write locks, choosing appropriate shard keys, and sharding as early as possible for future-proofing. Additionally, it compares MongoDB sharding to other systems like MySQL and Cassandra, highlighting challenges in rebalancing and schema changes.
Lightning Talk: What You Need to Know Before You Shard in 20 Minutes
- 1. Sharding in 20 minutes Why;Who;When;Where; David Murphy , Mongo Master Lead DBA, ObjectRocket @dmurphy_data @objectrocket
- 2. Background • 16 yrs in databases, development, & system engineering • Lead DBA @ ObjectRocket • Mongo Master with a focus on sharding, chunks, and scaling mongo beyond normal means.
- 3. What does a sharded cluster look like?
- 4. Why;Who;When;Where; i) Why do we shard? ii) Who should I shard? iii) When do we shard? iv) Where is my shard key
- 5. Why do we shard? • Scaling out write locks
- 6. Why do we shard? • Scaling out write locks • Small dataset to search per node
- 7. Why do we shard? • Scaling out write locks • Small dataset to search per node • Getting more connections to the data
- 8. Why do we shard? • Scaling out write locks • Small dataset to search per node • Getting more connections to the data • More smaller node vs Scaling up to expensive nodes
- 9. Who should I shard? • Biggest Collections by Size
- 10. Who should I shard? • Biggest Collections by Size • Busiest Collection by changes
- 11. Who should I shard? • Biggest Collections by Size • Busiest Collection by changes • Groupings of data (example): • State/Country • UserID • Company • Category
- 12. When do I shard? ALWAYS as early as possible!
- 13. When do I shard? ALWAYS as early as possible! Reasons: • Not all commands work
- 14. When do I shard? ALWAYS as early as possible! Reasons: • Not all commands work • Future Proof - No recoding
- 15. When do I shard? ALWAYS as early as possible! Reasons: • Not all commands work • Future Proof - No recoding • Adding index once your live can take time you don’t have!
- 16. Where (and what) is my shard key? You have to pick your own :/ But there are some quick hints…
- 17. Where (and what) is my shard key? Sharding Quick Hints: • Hashed Shard keys Great for even disk usage Uses Scatter-Gathers == More Conns Dates,Increasing IDs , and text are great here
- 18. Where (and what) is my shard key? Sharding Quick Hints: • Hashed Shard keys Great for even disk usage Uses Scatter-Gathers == More Conns Dates,Increasing IDs , and text are great here • Non-Hashed Keys Use profiler_level:2 & review ALL operations Things you wont change only No Dates No Increasing ID numbers No Text
- 19. Why mongo sharding/balancing Modulus Sharding with MySQL: Hard to rebalance online Requires application coding to support Ring Topologies like Cassandra - Cant change schema online Hard to rebalance online
- 20. Further Reading Presentations: Kenny Gorman Sharding - bit.ly/1oXYDfm David Murphy - Adv Sharding for Operations - bit.ly/1oXYDfm Other Sharding MongoDB Links - bit.ly/ZTtDI1 Picking a shard key (manual) - http://bit.ly/1ozuzMH Choosing a shard key - http://slidesha.re/1nBnGtq
- 21. Contact @dmurphy_data @objectrocket david@objectrocket.com https://www.objectrocket.com WE ARE HIRING! (DBA,DEVOPS, and more) https://www.objectrocket.com/careers