MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive
1 like273 views
The document discusses MongoDB Atlas's data lake capabilities, highlighting the challenges businesses face with managing vast amounts of data projected to reach 175 zettabytes by 2025. It outlines the architecture and security of Atlas, detailing how it emulates MongoDB functionalities while providing users control over their data and access configurations. The document also mentions various file formats supported and future optimizations, as well as potential integrations with cloud services like Microsoft Azure and Google Cloud.
![Configuration: Data
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive-mdbhouston19-190819145737/85/MongoDB-local-Houston-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-17-320.jpg)
![Configuration: Data (Future)
history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
db: "customers",
collection: "invoices"
}]
}](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive-mdbhouston19-190819145737/85/MongoDB-local-Houston-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-18-320.jpg)
![Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "$amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive-mdbhouston19-190819145737/85/MongoDB-local-Houston-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-24-320.jpg)
MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 2. Atlas Data Lake Technical Deep-Dive Subhead for the presentation goes here Craig Wilson, Senior Staff Engineer, MongoDB
- 3. State of Affairs Businesses have a humongous amount of data • IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public cloud. Cloud storage is cost-effective Cloud storage is hard to operationalize
- 4. A New Service Offered by MongoDB Atlas Access long-term data Query long-term data Analyze long-term data
- 5. Requirements Look and act like MongoDB Access customer’s data securely Handle queries over vast amounts of data Handle long-running queries Efficient use of resources
- 7. Language Must be able to communicate with our drivers Written in Go Implemented a TCP server Used mongo-go-driver’s wireprotocol package Used mongo-go-driver's bson package
- 8. Security Must have the same security as MongoDB Users configured in Atlas Implemented MongoDB’s security model Require the use of TLS + SNI(Server Name Indicator)
- 9. Behavior Must behave like MongoDB Implemented commands for a read-only server Used the server’s aggregation engine
- 10. Customer’s Data
- 11. Security: Customers Customers have complete control Provide us with an IAM Role Configure your buckets Configure your users in Atlas
- 12. Security: Atlas Atlas controls access to your data Storage of IAM Role Temporary Credentials
- 13. Configuration Customers control their data layout Stores Databases, Collections DataSources CollectionCollection Store Store DataSource DataSource DataSource
- 14. Configuration: File Formats • BSON (gzipped) • JSON (gzipped) • Avro (gzipped) • CSV/TSV (gzipped) • Parquet • XLSX
- 15. Configuration (S3 Bucket): ent-archive /archive/customers - a-m.json - n-z.json /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 16. Configuration: Store s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-east-1", prefix: "/archive/" }
- 17. Configuration: Data history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] }
- 18. Configuration: Data (Future) history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}.json.gz }, { store: "atlas", db: "customers", collection: "invoices" }] }
- 19. Queries
- 20. Processing MQL à Distributed MQL Parse Parallelize Distribute
- 21. Architecture Atlas Control Control Plane Compute Plane Data Plane DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer
- 23. Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 } { $match: { year: { $gt: 2000 } } } { $limit: 10 }
- 24. Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "$amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } } { $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
- 25. Future
- 26. Future More supported MongoDB operators. $out $merge Geo operators Full Text Search
- 31. Craig Wilson Senior Staff Engineer, MongoDB
- 32. Thank You!