MongoDB World 2019: MongoDB Atlas Data Lake Technical Deep Dive
1 like441 views
The document provides an overview of MongoDB's Atlas Data Lake, which addresses the challenges of managing large volumes of data and the need for cost-effective cloud storage. It highlights features such as secure access to long-term data, compatibility with MongoDB's language and security models, and various configurations for data storage and processing. Additionally, the document outlines future enhancements, including support for more file formats and operators, as well as potential integrations with major cloud providers.
![history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}
Configuration: Data](https://image.slidesharecdn.com/suttonnday21245pmatlasdatalaketechnicaldeepdive-190701172350/85/MongoDB-World-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-16-320.jpg)
![history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
cluster: "my-cluster",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)](https://image.slidesharecdn.com/suttonnday21245pmatlasdatalaketechnicaldeepdive-190701172350/85/MongoDB-World-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-17-320.jpg)
![{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }](https://image.slidesharecdn.com/suttonnday21245pmatlasdatalaketechnicaldeepdive-190701172350/85/MongoDB-World-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-23-320.jpg)
MongoDB World 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 1. Craig Wilson – Software Engineer Atlas Data Lake Technical Deep-Dive @craiggwilson
- 2. State of Affairs Why are we building this? • Businesses have a humongous amount of data • IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public cloud. • Cloud storage is cost-effective • Cloud storage is hard to operationalize
- 3. A New Service Offered by MongoDB Atlas Atlas Data Lake allows you to... § Access long-term data § Query long-term data § Analyze long-term data
- 4. Requirements Every product has requirements! § Look and act like MongoDB § Access customer’s data securely § Handle queries over vast amounts of data § Handle long-running queries § Efficient use of resources
- 6. Language Must be able to communicate with our drivers. § Written in Go § Implemented a TCP server § Used mongo-go-driver’s wireprotocol package § Used mongo-go-driver's bson package
- 7. Security Must have the same security as MongoDB. § Users configured in Atlas § Implemented MongoDB’s security model § Authentication § Authorization § Require the use of TLS + SNI § SNI = Server Name Indicator
- 8. Behavior Must behave like MongoDB. § Implemented commands for a read-only server § Used the server’s aggregation engine
- 10. Security: Customers Customers have complete control. § Provide us with an IAM Role § Configure your buckets § Configure your users in Atlas
- 11. Security: Atlas Atlas controls access to your data. § Storage of IAM Role § Temporary Credentials
- 12. Configuration Customers control their data layout. § Stores § Databases, Collections § DataSources CollectionCollection Store Store Database DataSource DataSource DataSource
- 13. Configuration: File Formats Each file has a format. § BSON (gzipped) § JSON (gzipped) § Avro (gzipped) § CSV/TSV (gzipped) § Parquet
- 14. /archive/customers - a-m.json - n-z.json Configuration (S3 Bucket): ent-archive /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 15. s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-east-1", prefix: "/archive/" } Configuration: Store
- 16. history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] } Configuration: Data
- 17. history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}.json.gz }, { store: "atlas", cluster: "my-cluster", db: "customers", collection: "invoices" }] } Configuration: Data (Future)
- 18. Queries
- 19. Processing MQL à Distributed MQL § Parse § Parallelize § Distribute
- 20. Architecture Atlas Control Control Plane Compute Plane Data Plane DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer
- 22. { $match: { year: { $gt: 2000 } } } { $limit: 10 } Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 }
- 23. { $group: { _id: "$year", totalAvg: { $avg: "amount" } } } Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
- 24. Future
- 25. Future More supported MongoDB operators. § $out § $merge § $graphLookup § Geo operators § Full Text Search
- 27. Future More File Formats! § ORC § Excel § PDF
- 28. Future Integrations! § Atlas § Microsoft Azure § Google Cloud
- 29. Hiring Lots to do! § mongodb.com/careers