MongoDB .local Bengaluru 2019: MongoDB Atlas Data Lake Technical Deep Dive
0 likes446 views
The document details the MongoDB Atlas Data Lake, a new service designed to help businesses manage and analyze vast amounts of long-term data, which is becoming increasingly important as global data reaches 175 zettabytes by 2025. It outlines the product's requirements, configuration options, and security measures, emphasizing customer control over data layout and access. The document also presents architecture examples, potential future optimizations, and additional file format support and integrations.
![{
“stores” : [{
s3 : {
name: <string>,
bucket: <string>,
region: <string>,
prefix: <string>
}}
],
“databases” : {
“<database_name>” : {
“<collection_name>” : [
{
“store” : “<string>”
“definition” : “<string>”
}]
}}
}
Configuration: Store](https://image.slidesharecdn.com/datalaketechnicaldeepdivev1-190911204022/85/MongoDB-local-Bengaluru-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-13-320.jpg)
![databases : {
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: “ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}
}
Configuration: Data](https://image.slidesharecdn.com/datalaketechnicaldeepdivev1-190911204022/85/MongoDB-local-Bengaluru-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-16-320.jpg)
![history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
cluster: "my-cluster",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)](https://image.slidesharecdn.com/datalaketechnicaldeepdivev1-190911204022/85/MongoDB-local-Bengaluru-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-17-320.jpg)
![{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "amount" },
totalAvg_count: { $sum: 1 }} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum",
"$totalAvg_count"] } } }
Query Example: $group](https://image.slidesharecdn.com/datalaketechnicaldeepdivev1-190911204022/85/MongoDB-local-Bengaluru-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-23-320.jpg)
MongoDB .local Bengaluru 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 2. Atlas Data Lake Technical Deep-Dive Himanshumali, Solutions Architect-APAC, MongoDB
- 3. Himanshumali
- 4. State of Affairs Why are we building this? • Businesses have a humongous amount of data o IDC predicts that by 2025 global data will reach 175 Zettabytes o 49% of it will reside in the public cloud. • Cloud storage is cost-effective • Stored data is hard to operationalize
- 5. A New Service Offered by MongoDB Atlas Atlas Data Lake allows you to... • Access long-term data • Query long-term data • Analyze long-term data
- 6. Product Guidelines Every product has requirements! • Look and act like MongoDB • Access customer’s data securely • Handle queries over vast amounts of data • Handle long-running queries • Efficient use of resources
- 8. Behaviour • Communicate with our existing drivers. Written in Go. • Implemented a TCP server, using mongo-go-driver’s wire protocol package • Implemented commands for a read-only server • Used the server’s aggregation engine
- 9. Security Must have the same security as MongoDB. • Users configured in Atlas • Implemented MongoDB’s security model • Authentication • Authorization • Require the use of TLS + SNI • SNI = Server Name Indicator
- 11. Controlled by Customer Customers have complete control • Provide us with an IAM Role • Configure your buckets • Configure your users in Atlas
- 12. Configuration Customers control their data layout. • Stores • Databases, Collections • DataSources
- 13. { “stores” : [{ s3 : { name: <string>, bucket: <string>, region: <string>, prefix: <string> }} ], “databases” : { “<database_name>” : { “<collection_name>” : [ { “store” : “<string>” “definition” : “<string>” }] }} } Configuration: Store
- 14. Configuration (S3 Bucket): ent-archive /archive/customers - a-m.json - n-z.json /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 15. s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-east-1", prefix: "/archive/" } Configuration: Store
- 16. databases : { history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: “ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] } } Configuration: Data
- 17. history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}.json.gz }, { store: "atlas", cluster: "my-cluster", db: "customers", collection: "invoices" }] } Configuration: Data (Future)
- 18. Configuration: File Formats Each file has a format • BSON (gzipped) • JSON (gzipped) • Avro (gzipped) • CSV/TSV (gzipped) • Parquet
- 19. Processing MQL : Distributed MQL • Parse • Distribute & Parallelize
- 20. Architecture
- 21. Architecture
- 22. { $match: { year: { $gt: 2000 } } } { $limit: 10 } Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 } Query Example: $limit
- 23. { $group: { _id: "$year", totalAvg: { $avg: "amount" } } } Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "amount" }, totalAvg_count: { $sum: 1 }} } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" }} } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } } Query Example: $group
- 24. Use Cases Scenario • Customer storing devices data in Atlas for last 2 years • Actively requires on last 2 years of data • Might also need to query older data in case of customer request
- 25. Future
- 26. Future More supported MongoDB operators. • $out • $merge • $graphLookup • Geo operators • Full Text Search
- 28. Future More File Formats! • ORC • Excel • PDF
- 29. Future Integrations! • Microsoft Azure • Google Cloud
- 30. Thank You!