MongoDB World 2018: Load Spikes, GDPR, & Natural Disasters... Oh My! Scaling an App with MongoDB Atlas
0 likes236 views
This document discusses how MongoDB Atlas can help scale an application to handle load spikes, data privacy regulations like GDPR, and natural disasters. It begins with an overview of MongoDB Atlas and its features like easy deployment, health monitoring, automation and migration. It then demonstrates how to create a free tier cluster and shows the limitations of the free tier like storage size, connections and operations limits. The document discusses upgrading from the free tier to a paid cluster and setting up cross-region replication for disaster recovery. It also covers using indexes, monitoring performance with Atlas advisor, and scaling resources like upgrading from M10 to M20 instances. The latter part discusses sharding to improve latency for international users, setting up zones for data locality, and enabling
MongoDB World 2018: Load Spikes, GDPR, & Natural Disasters... Oh My! Scaling an App with MongoDB Atlas
- 2. Load Spikes, GDPR, and Natural Disasters…. Oh My! Scaling an App With Atlas
- 3. IAN BOROS Atlas Enterprise Software Engineer MongoDB, Inc Server Query Team Software Engineer MongoDB, Inc FIONA ROWAN
- 9. ● User metadata ● Image metadata ● Friends ● Posts
- 10. ● User metadata ● Image metadata ● Friends ● Posts I think we need a database
- 11. ● User metadata ● Image metadata ● Friends ● Posts I think we need a database Is this the “Cloud”?
- 12. What is MongoDB Atlas? MongoDB Atlas is our Database as a Service ● Easy deployment ● Health monitoring ● Automation ● Migration DATABASEAPPLICATION
- 13. What is MongoDB Atlas? MongoDB Atlas is our Database as a Service ● Easy deployment ● Health monitoring ● Automation ● Migration DATABASEAPPLICATION
- 14. Close up of the Database Typically a “Replica Set” Secondaries replicate data for durability and consistency Primaries accept writes
- 15. Get Started FREE Create an account and click “Create Cluster” Choose instance size M0
- 16. Look under your chairs everybody...
- 17. Look under your chairs everybody... FREE CLOUD DATABASES FOR EVERYONE!!!
- 18. Demo: Atlas Free Tier
- 26. Atlas Free Tier Limitations
- 27. Close up of the Free Tier Database CLIENT 1 CLIENT 3CLIENT 2 CLIENT 1 CLIENT 2 CLIENT 3 CLIENT 1 CLIENT 2 CLIENT 3
- 28. The Free Tier is actually many clients using the SAME database on the SAME machine. All operations go through a Proxy Atlas Free Tier Limitations PROXY APPLICATION APPLICATION APPLICATION
- 29. Metrics
- 30. Connections
- 31. Data Transfer
- 33. Have you outgrown the Free Tier? Free Tier maximums ❏ 100 operations / second ❏ 10GB / week of data transfer ❏ 512 MB of storage ❏ 100 connections ❏ 100 databases ❏ 500 collections
- 37. CLIENT 1 CLIENT 1 CLIENT 2 CLIENT 3 CLIENT 1 CLIENT 2 CLIENT 3 CLIENT 1 CLIENT 2 CLIENT 3 CLIENT 1 CLIENT 1 Multi-tenant cluster Single-tenant cluster
- 38. Demo: Upgrading to an M10
- 59. 2018 2022 Prototype Small userbase 2026 Large userbase in North America
- 65. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’
- 66. Collection Scans {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ db.users.findOne({username: “alice”})
- 67. Collection Scans {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ db.users.findOne({username: “alice”})
- 68. Collection Scans {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ db.users.findOne({username: “alice”})
- 69. Collection Scans {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ db.users.findOne({username: “alice”})
- 70. Collection Scans {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ db.users.findOne({username: “alice”})
- 71. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 72. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 73. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 74. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 75. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 76. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 77. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 78. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 79. Indexes {_id: 0, username: “bob”, ...} {_id: 1, username: “samuel”, ...} {_id: 2, username: “ian”, ...} {_id: 3, username: “fiona”, ...} {_id: 4, username: “alice”, ...} ‘users’ “alice” “bob” “fiona” “ian” “samuel” {username: 1} index Sorted
- 80. Queries DATABASEAPPLICATION db.users.find({username: ___}) db.posts.find({username: ___, creation_date: ___}) .sort({creation_date: -1}) db.messages.find({fromUser: ___, toUser: ___});
- 86. RAM Usage
- 87. ❏ Normalized CPU is near 100% ❏ Disk Utilization is near 100% ❏ RAM Utilization is near 100% (most common bottleneck) ❏ Queues of Operations is growing ❏ Operation Execution Times is growing Look out for these Metrics
- 88. Demo: M10 -> M20
- 90. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon
- 95. Latency Users in France are reporting slow reads and writes!
- 97. What is sharding? MONGOS CONFIG SERVER SHARD 1 SHARD 2 APPLICATION Reads and Writes
- 98. Shard Keys & Chunks {userId: MinKey} {userId: MaxKey} {userId: 0} {userId: 150} {userId: -100} Shard 1 Shard 2 Shard 1 Shard 2
- 99. Intro To Global Writes
- 100. SHARD 1
- 101. SHARD 1 SHARD 2
- 102. SHARD 1 SHARD 2
- 103. SHARD 1 SHARD 2
- 104. SHARD 1 SHARD 2 DATABASE This shard is in France, and users closest to France have their data stored in this shard This shard is in N. Virginia, users closest to North America have their data stored in this shard Shard Key: {location: 1, userId: 1} Intro To Global Writes
- 105. Zones {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey}
- 106. Zones North AmericaZone 1: {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey}
- 107. Zones {location: “Canada”, userId: MinKey} {location: “Canada”, userId: MaxKey} North AmericaZone 1: {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey}
- 108. Zones {location: “Canada”, userId: MinKey} {location: “Canada”, userId: MaxKey} North AmericaZone 1: {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey} {location: “US”, userId: MinKey} {location: “US”, userId: MaxKey}
- 109. Zones {location: “Canada”, userId: MinKey} {location: “Canada”, userId: MaxKey} North AmericaZone 1: EuropeZone 2: {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey} {location: “US”, userId: MinKey} {location: “US”, userId: MaxKey}
- 110. Zones {location: “Canada”, userId: MinKey} {location: “Canada”, userId: MaxKey} North AmericaZone 1: EuropeZone 2: {location: MinKey, userId: MinKey} {location: MaxKey, userId: MaxKey} {location: “France”, userId: MaxKey} {location: “France”, userId: MaxKey} {location: “US”, userId: MinKey} {location: “US”, userId: MaxKey}
- 111. MONGOS CONFIG SERVER SHARD 1 SHARD 2 APPLICATION Reads and Writes
- 112. MONGOS CONFIG SERVER SHARD 1 SHARD 2 APPLICATION Reads and Writes North America
- 113. MONGOS CONFIG SERVER SHARD 1 SHARD 2 APPLICATION Reads and Writes North America Europe
- 114. SHARD 1 SHARD 2
- 115. How do we turn a Replica Set into a Sharded Cluster?
- 120. MONGOS CONFIG SERVER SHARD 1 SHARD 2
- 121. MONGOS CONFIG SERVER SHARD 1 SHARD 2Chunk Migrations
- 122. Demo: Global Writes
- 125. Data Privacy Regulations General Data Protection Regulation recently went into effect
- 127. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon
- 128. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon No money
- 129. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon Overwhelming Load No money
- 130. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon Overwhelming Load No money Data outage
- 131. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon Overwhelming Load Overwhelming Load No money Data outage
- 132. 2018 2022 Prototype Small userbase 2026 Large userbase in North America 2030 International Phenomenon Overwhelming Load Overwhelming Load Slow reads and writes overseas No money Data outage
- 133. Concluding Statements We don’t want you to be killed by success In ten minutes from now you can have a cluster running. There’s no excuse for you not to be developing that app of yours