A Technical Introduction to WiredTiger

Editor's Notes

• #3: Feel free to ask questions as we go, hopefully there will be a few minutes for Q&A at the end. Also happy to discuss by email, let me know how we can help.
• #5: Build a toolkit: one path is to build special-purpose engines to handle specific workloads, another path is to handle complex/changing workloads.
• #8: This kind of positive feedback isn’t common in engineering groups.
• #10: The structure for the rest of the talk.
• #11: Traditional storage engine designs struggle with modern hardware. I/O, in relationship to memory, a worse outcome than ever before: trade CPU for I/O wherever possible. Big block I/O: if we have to do I/O, bring in a lot of data.
• #12: Moderately complex inside. Outside APIs, handle + method Top-level schema layer, where every table and associated indexes Operations are transactionally protected, implemented in terms of in-memory snapshots. Operations are to a row-store engine (ordered key/value pair), column-store (key is a 64-bit record number) Block management is intended to be pluggable itself. On disk, key/value pair files and log files.
• #13: From now on, I’m going to mostly be talking about “trees” in-memory, without distinguishing what type of tree – here’s the overview, after which it’s just a page in-memory.
• #14: WiredTiger focuses on in-memory performance: I/O means you’ve lost the war, you’re only trying to retreat gracefully.
• #15: WiredTiger does have root pages and internal pages, with leaf pages at the bottom: binary search of each page yields the child page for the subsequent search. Importantly, pointers in memory are not disk offsets (in many engines in-memory objects find each other use disk offsets, so, for example, a transition from an internal page to a leaf page implies giving the cache subsystem a disk offset, and the cache returns the in-memory pointer, possibly after a read. The WiredTiger in-memory tree is exactly that, it’s an in-memory optimized tree. This is good (fast in-memory operations), this is bad (we have a translation step after reading, and before writing, disk blocks). We knew we wanted to modify our in-memory representation without changing our on-disk format (avoid upgrades!), and we knew a lot of our compression algorithms would require translation before writing anyway (for example, our on-disk pages have no indexing information, saves about 15-20% of the disk footprint in some workloads).
• #16: To make this efficient, pointers need to be protected: once data is larger than cache, there needs to be a check to ensure a pointer is valid. There’s a background thread doing eviction of pages. Hazard pointers can be thought of as micro-logging. Readers/writers publishes the memory address of a page it wants on a non-shared cache line; after that publish, if the pointer is still valid, can proceed. Eviction threads must check those locations to ensure a page is not currently in use: eviction bears the burden, readers/writers go fast. Design principal: application threads never wait, shift work from application threads to system-internal threads.
• #17: Different in-memory representation vs on-disk – opportunities to optimize plus some interesting features. Read on-disk page into cache, generally a read-only image. Add indexing information (binary search) on top of that image. Updates are layered on top of that image, including new key/value pairs inserted between existing keys. if the page grows too large, background threads will deal with it. Lots of magic in traversal, threads must go back-and-forth between the original image and the updated image. Writing a page: Combine the previous page image with in-memory changes If the result is too big, split Allocate new space in the file Always write multiples of the allocation size
• #19: To summarize: Note we haven’t talked about locking at all: application threads can retrieve and update data without every acquiring a lock. Justin Levandoski’s BW-Tree work: they’ve avoided taking pages out of circulation during splits, interesting.
• #21: Readers don’t block writers, writers don’t block readers, writers don’t block writers, again, no locks have been acquired.
• #22: If there have been no updates, then it’s easy, the on-page item is the correct item for any query. If there are updates, each update has associated with it a transaction ID, and that transaction ID combined with the transaction’s snapshot, determines the correct value.
• #23: Index references the original page image, once updates installed readers/writers have to check for updates. The first update in the list is generally the one we want, if it’s not yet visible, other updates are checked. if no update is correct, the value on the original page must be the one we want. All updates done by atomic updates, swapping a new pointer into place, readers concurrent to updates. Writing the page to disk requires processing all of this information, and determining the values to write.
• #26: Page-Write: transform the in-memory version: selecting values to write, page-splitting, all sorts of compression, checksums. Checkpoints are simply another “snapshot reader”, so they run concurrently with other readers and writers. Writing a page: Combine the previous page image with in-memory changes Allocate new space in the file If the result is too big, split Always write multiples of the allocation size Can configure direct I/O to keep reads and writes out of the filesystem cache
• #27: All of these apply in-memory and on disk, so we save both on disk and in the cache.
• #28: in the same code paths, we compress the data.
• #29: In WiredTiger, compact file sizes, and that certain types of compression cannot be turned off, WiredTiger without “compression”, is 50%.
• #31: Storage engines are all about not losing your stuff. Pretty standard WAL implementation: before a commit is visible, a log record with all of the changes in the transaction has been flushed to stable storage. Group commit: concurrent log writes are done with a single storage flush. Started with “Scalability of write-ahead logging on multicore and multisocket hardware by Ryan Johnson, Ippokratis Pandis, Radu Stoica, Manos Athanassoulis and Anastasia Ailamaki.”, and there’s lots more engineering there.
• #32: Checkpoints move from one stable point to another one. Our goal was to build a single-node engine, and for that reason, we had to run without a log without losing durability.
• #35: Lookaside tables.
• #37: A very large data-set over time: blue: a btree tails off over time as the probability of a page being found in cache decreases (that’s why the random nature of the insert matters). red/green flatter: only maintain the recent updates in cache, and merge the updates in the background. LSM is write-optimized, though: the reason the btree is primary is that read-mostly workloads generally behave better in a btree than in LSM. What we want to do eventually, is enable the conversion of an LSM tree into a btree (if you think of a forest of btrees collapsing into a single btree, that matches nicely with the typical workload of inserting a lot of data and then processing that data. Ideally, we’d also be able to reverse that process on demand.