MongoDB .local Chicago 2019: MongoDB Atlas Data Lake Technical Deep Dive
1 like237 views
The document discusses the MongoDB Atlas Data Lake, emphasizing its ability to handle large volumes of data while ensuring security and efficient querying. It highlights the requirements for integration with MongoDB, such as secure access and operational capabilities, and outlines various data formats and configurations available for customers. Future developments include additional supported MongoDB operators, optimizations, and integrations with cloud services.
![Configuration: Data
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive20191021craigchicago-191025171245/85/MongoDB-local-Chicago-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-16-320.jpg)
![history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive20191021craigchicago-191025171245/85/MongoDB-local-Chicago-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-17-320.jpg)
![Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "$amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }](https://image.slidesharecdn.com/atlasdatalaketechnicaldeepdive20191021craigchicago-191025171245/85/MongoDB-local-Chicago-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-23-320.jpg)
MongoDB .local Chicago 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 1. #MDBlocal Atlas Data Lake Technical Deep-Dive CHICAGO Craig Wilson, Senior Staff Engineer, MongoDB
- 2. #MDBLocal State of Affairs Businesses have a humongous amount of data • IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public cloud. Cloud storage is cost-effective Cloud storage is hard to operationalize
- 3. #MDBLocal A New Service Offered by MongoDB Atlas Access long-term data Query long-term data Analyze long-term data
- 4. #MDBLocal Requirements Look and act like MongoDB Access customer’s data securely Handle queries over vast amounts of data Handle long-running queries Efficient use of resources
- 6. #MDBLocal Language Must be able to communicate with our drivers Written in Go Implemented a TCP server Used mongo-go-driver’s wireprotocol package Used mongo-go-driver's bson package
- 7. #MDBLocal Security Must have the same security as MongoDB Users configured in Atlas Implemented MongoDB’s security model Require the use of TLS + SNI(Server Name Indicator)
- 8. #MDBLocal Behavior Must behave like MongoDB Implemented commands for a read-only server Used the server’s aggregation engine
- 10. #MDBLocal Security: Customers Customers have complete control Provide us with an IAM Role Configure your buckets Configure your users in Atlas
- 11. #MDBLocal Security: Atlas Atlas controls access to your data Storage of IAM Role Temporary Credentials
- 12. #MDBLocal Configuration Customers control their data layout Stores Databases, Collections DataSources CollectionCollection Store Store DataSource DataSource DataSource
- 13. #MDBLocal Configuration: File Formats • BSON (gzipped) • JSON (gzipped) • Avro (gzipped) • CSV/TSV (gzipped) • Parquet • XLSX
- 14. Configuration (S3 Bucket): ent-archive /archive/customers - a-m.json - n-z.json /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 15. Configuration: Store s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-east-1", prefix: "/archive/" }
- 16. Configuration: Data history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] }
- 17. history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}.json.gz }, { store: "atlas", db: "customers", collection: "invoices" }] } Configuration: Data (Future)
- 18. Queries
- 19. #MDBLocal Processing MQL à Distributed MQL Parse Parallelize Distribute
- 20. #MDBLocal Architecture Atlas Control Control Plane Compute Plane Data Plane DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer
- 22. Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 } { $match: { year: { $gt: 2000 } } } { $limit: 10 }
- 23. Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "$amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } } { $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
- 24. Future
- 25. #MDBLocal Future More supported MongoDB operators. $out $merge Geo operators Full Text Search
- 27. #MDBLocal Future File Formats • ORC • PDF Compression • Bzip2 • Snappy • LZMA • LZO • Zstd
- 30. Craig Wilson Senior Staff Engineer, MongoDB
- 31. Our Developer focused talks are back on the road! Find one near you At your MongoDB.local, you’ll learn technologies, tool, and best practices That make it easy for you to build data-driven applications without distraction.
- 32. THANK YOU