MongoDB Days Silicon Valley: A Technical Introduction to WiredTiger
Download as POTX, PDF0 likes1,208 views
The document details the integration of WiredTiger storage engine into MongoDB, highlighting its benefits such as improved concurrency, compression capabilities, and efficient caching. It compares performance metrics of WiredTiger against the older mmapv1 engine, showcasing significant improvements in operations per second and reduced storage size. Overall, WiredTiger is positioned as a more advanced and tunable option for diverse database use cases.
![Sample Document
{
"_id" : ObjectId("5553a98ce4b02cf7150dee1f"),
"st" : "x+50600-002700",
"ts" : ISODate("1984-01-01T00:00:00Z"),
"position" : {
"type" : "Point",
"coordinates" : [
-2.7,
50.6
]
},
"elevation" : 9999,
"callLetters" : "BUOY",
"qualityControlProcess" : "V020",
"dataSource" : "4",
"type" : "FM-13",
"airTemperature" : {
"value" : 999.9,
"quality" : "9"
},
"dewPoint" : {
"value" : 999.9,
"quality" : "9"
},
"pressure" : {
"value" : 9999.9,
"quality" : "9"
},
"wind" : {
"direction" : {
"angle" : 999,
"quality" : "9"
},
"type" : "9",
"speed" : {
"rate" : 999.9,
"quality" : "9"
}
},
"visibility" : {
"distance" : {
"value" : 999999,
"quality" : "9"
},"variability" : {
"value" : "N",
"quality" : "9"}},
"skyCondition" : {
"ceilingHeight" : {
"value" : 99999,
"quality" : "9",
"determination" : "9"},
"cavok" : "N"},
"sections" : [
"AG1"
],
"precipitationEstimatedObservation" : {
"discrepancy" : "2",
"estimatedWaterDepth" : 0
}
}](https://image.slidesharecdn.com/wt-integration-updated-151204012448-lva1-app6892/85/MongoDB-Days-Silicon-Valley-A-Technical-Introduction-to-WiredTiger-19-320.jpg)
MongoDB Days Silicon Valley: A Technical Introduction to WiredTiger
- 2. WiredTiger MongoDB Integration By: Osmar “Oz” Olivo Product Manager MongoDB Inc.
- 3. Agenda • WiredTiger Integration • Document Level Locking • Compression • Caching
- 5. • Vision – One data model, one API, one set of operational concerns » Under the hood, many options for every use case under the sun Storage Engine Layer
- 6. Storage Engine Layer Content Repo IoT Sensor Backend Ad Service Customer Analytics Archive MongoDB Query Language (MQL) + Native Drivers MongoDB Document Data Model MMAP V1 WT In-Memory ? ? Future Possible Storage Engines Management Security Beta in MongoDB 3.2
- 9. Why WT? • WT product and team acquired by MongoDB – Authors former members of Berkeley DB team – Standalone engine in use in production today • WT addresses weaknesses of MMAPv1 – Compression – Highly concurrent and enables full hardware utilization – More tunable
- 11. Document Level Concurrency • Improved Concurrency – Uses various algorithms to minimize contention between threads – One thread yields on contention to same document • Writes no longer block all other writes • CPU utilization directly correlates with performance
- 12. Concurrency Model MMAPv1 WiredTiger W W R R R W R W RW
- 13. Relative Comparisons 5M Operations Avg. Record size = 1k 10 Threads Batch Size = 1000 12 Cores SSD’s Acknowledge Writes
- 14. WT • 5M Documents Inserted • RunTime: 46.3 seconds • Throughput(ops/sec): 107,972.7 *Saturated SSD’s at ~2.25x MMAP • 5M Documents Inserted • RunTime: 103.7 seconds • Throughput(ops/sec): 48,213.2 Insert Only
- 15. WT • Docs updated randomly 5M times • RunTime:41.7 seconds • Throughput(ops/sec): 119,817 *3.3x Faster MMAP • Docs updated randomly 5M times • RunTime: 139.3 seconds • Throughput(ops/sec): 35,903 Update Many
- 16. Update One WT • 1 Document updated 5M times • RunTime: 99.97 seconds • Throughput(ops/sec): 50,014 MMAP • 1 Document updated 5M times • RunTime: 90 seconds • Throughput(ops/sec): 55,625
- 17. Compression
- 18. Compression (On by default) • WT supports two compression algorithms – snappy ( default) • Good compression benefits with little CPU/performance impact – zlib • Extremely good compression at a cost of additional CPU/degraded performance – Prefix compression • Used for Indexes (on by default)
- 19. Sample Document { "_id" : ObjectId("5553a98ce4b02cf7150dee1f"), "st" : "x+50600-002700", "ts" : ISODate("1984-01-01T00:00:00Z"), "position" : { "type" : "Point", "coordinates" : [ -2.7, 50.6 ] }, "elevation" : 9999, "callLetters" : "BUOY", "qualityControlProcess" : "V020", "dataSource" : "4", "type" : "FM-13", "airTemperature" : { "value" : 999.9, "quality" : "9" }, "dewPoint" : { "value" : 999.9, "quality" : "9" }, "pressure" : { "value" : 9999.9, "quality" : "9" }, "wind" : { "direction" : { "angle" : 999, "quality" : "9" }, "type" : "9", "speed" : { "rate" : 999.9, "quality" : "9" } }, "visibility" : { "distance" : { "value" : 999999, "quality" : "9" },"variability" : { "value" : "N", "quality" : "9"}}, "skyCondition" : { "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9"}, "cavok" : "N"}, "sections" : [ "AG1" ], "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 0 } }
- 20. MMAPv1 "ns" : "ncdc.data", "count" : 33382365, "size" : 58GB, "avgObjSize" : 1887 Bytes, "numExtents" : 51, "storageSize" : 60GB, "nindexes" : 1, "totalIndexSize" : 1400MB, "indexSizes" : { "_id_" : 1400MB } WT (Snappy) "ns" : "ncdc.data", "count" : 33382365, "size" : 44GB, "avgObjSize" : 1419 Bytes, "storageSize" : 8GB, "nindexes" : 1, "totalIndexSize" : 289MB, "indexSizes" : { "_id_" : 289MB } Data Size
- 21. WT (ZLIB) "ns" : "ncdc.data", "count" : 33382365, "size" : 44GB, "avgObjSize" : 1419 Bytes, "storageSize" : 3GB, "nindexes" : 1, "totalIndexSize" : 289MB, "indexSizes" : { "_id_" : 289MB }, Data Size
- 22. Caching
- 24. Caching • WT cache should be tuned to fit as much of the working set as possible. • When should I tune the WT cache down?
- 25. In Summary • The 3 C’s of the WiredTiger Storage Engine – Concurrency – Compression – Caching
- 26. Transparent To The Application Content Repo IoT Sensor Backend Ad Service Customer Analytics Archive MongoDB Query Language (MQL) + Native Drivers MongoDB Document Data Model MMAP V1 WT In-Memory ? ? Supported in MongoDB Future Possible Storage Engines Management Security Beta in MongoDB 3.2
- 28. Questions?
- 29. #MDBDays mongodb.com Get your technical questions answered In the foyer, 10:00 - 5:00 By appointment only – register in person
- 30. Tell me how I did today on Guidebook and enter for a chance to win one of these How to do it: Download the Guidebook App Search for MongoDB Silicon Valley Submit session feedback