Linux Kernel Extension for Databases / Александр Крижановский (Tempesta Technologies)
1 like1,361 views
The document discusses the development of Linux kernel extensions for databases by Tempesta Technologies, emphasizing the inefficiencies of bypassing the OS for database management. It highlights various lessons learned regarding memory management, filesystem structures, and the importance of data consistency controls. The content also introduces TempestaDB, an in-memory database optimized for performance and designed for use with a web-accelerator and firewall.
![mmap(2): index for free
$ grep 6f0000000000 /proc/[0-9]*/maps
$
DbItem *db = mmap(0x6f0000000000, 0x40000000 /* 1GB */, ...);
DbItem *x = (DbItem *)(0x6f0000000000 + key);](https://image.slidesharecdn.com/7-161114105950/85/Linux-Kernel-Extension-for-Databases-Tempesta-Technologies-10-320.jpg)
![...or just an array
DbItem *db = mmap(0, 0x40000000 /* 1GB */, ...);
DbItem *x = &db[key];](https://image.slidesharecdn.com/7-161114105950/85/Linux-Kernel-Extension-for-Databases-Tempesta-Technologies-11-320.jpg)
![Transactional filesystems: BtrFS
Uses log-trees, so [probaly] can be used instead of doublewrite buffer
ioctl(): BTRFS_IOC_TRANS_START and BTRFS_IOC_TRANS_END](https://image.slidesharecdn.com/7-161114105950/85/Linux-Kernel-Extension-for-Databases-Tempesta-Technologies-31-320.jpg)
Linux Kernel Extension for Databases / Александр Крижановский (Tempesta Technologies)
- 1. Linux Kernel Extensions for Databases Alexander Krizhanovsky Tempesta Technologies, Inc. ak@tempesta-tech.com
- 2. Who am I? CEO & CTO at NatSys Lab & Tempesta Technologies Tempesta Technologies (Seattle, WA) ● Subsidiary of NatSys Lab. developing Tempesta FW – a first and only hybrid of HTTP accelerator and firewall for DDoS mitigation & WAF NatSys Lab (Moscow, Russia) ● Custom software development in: – high performance network traffic processing – databases
- 3. The begin (many years ago) Database to store Instant messenger's history Plenty of data (NoSQL, 3-touple key) High performance Some consistency (no transactions) 2-3 months (quick prototype)
- 4. Simple DBMS Disclamer: memory & I/O only, no index, no locking, no queries “DBMS” means InnoDB
- 5. Linux VMM?
- 6. open(O_DIRECT): OS kernel bypass «In short, the whole "let's bypass the OS" notion is just fundamentally broken. It sounds simple, but it sounds simple only to an idiot who writes databases and doesn't even UNDERSTAND what an OS is meant to do.» Linus Torvalds «Re: O_DIRECT question» https://lkml.org/lkml/2007/1/11/129
- 7. mmap(2)! Automatic page eviction Transparrent persistency I/O is managed by OS ...and ever radix tree index for free!
- 8. x86-64 page table (radix tree)
- 9. A tree in the tree a.0 c.0 a.1 c.1 Page table a b c Application Tree
- 10. mmap(2): index for free $ grep 6f0000000000 /proc/[0-9]*/maps $ DbItem *db = mmap(0x6f0000000000, 0x40000000 /* 1GB */, ...); DbItem *x = (DbItem *)(0x6f0000000000 + key);
- 11. ...or just an array DbItem *db = mmap(0, 0x40000000 /* 1GB */, ...); DbItem *x = &db[key];
- 12. Virtual memory isn't for free TLB cache is small (~1024 entries, i.e. 4MB) TLB cache miss is up to 4 memory transfers Spacial locality is crucial: 1 address outlier is up to 12KB …but Linux VMM coalesces memory areas Context switch of user-space processes invalidates TLB ...but threads and user/kernel context switches are cheap
- 13. Lesson 1 Large mmap()'s are expensive Spacial locality is your friend Kernel mappings are resistant to context switches
- 14. DBMS vs OS Stonebreaker, "Operating System Support for Database Management”, 1981 ● OS buffers (pages) force out with controlled order ● Record-oriented FS (block != record) ● Data consistency control (transactions) ● FS blocks physical contiguity ● Tree structured FS: a tree in a tree ● Scheduling, process management and IPC
- 15. Filesystem: extents Modern Linux filesystems: BtrFS, EXT4, XFS Large contigous space is allocated at once Per-extent addressing
- 16. Lesson 2 There are no (or small) file blocks fragmentation There are no trees inside extent fallocate(2) became my new friend
- 17. Transactions and consistency control (InnoDB case)
- 18. Lesson 3 Atomicity: which pages and when are written Database operates on record granularity Q: Can modern filesystem do this for us?
- 19. Log-enhanced filesystems XFS – metadata only EXT4 – metadata and data Log writes are sequential, data updates are batched Double writes on data updates
- 20. Log-structured filesystems BSD LFS, Nilfs2 Dirty data blocks are written to next available segment Changed metadata is also written to new location ...so poor performance on data updates Inodes aren't at fixed location → inode map Garbage collection of dead blocks (with significant overhead) Poor fragmentation on large files → slow updates
- 21. Copy-On-Write filesystems BtrFS, ZFS Whole tree branches are COW'ed Constant root place Still fragmentation issues and heavy write loads Very poor at random writes (OLTP), better for OLAP
- 22. Soft Updates BSD UFS2 Proper ordering to keep filesystem structure consistent (metadata) Garbage collection to gather lost data blocks Knows about filesystem metadata, not about stored data
- 23. Lesson 4: Data consistency control Can log-enhanced data journaling FS replace doublewrite buffer? https://www.percona.com/blog/2015/06/17/update-on-the-innodb-double-write-buffer-and-ext4- transactions/ , by Yves Trudeau, Percona. NOT! ● Filesystem gurantees data block consistency, not group of blocks!
- 24. Lesson 5 Modern Linux filesystems are unstructured
- 25. Page eviction Typically current process reclaims memory kswapd – alloc_pages() slow path OOM active list inactive list add freeP P P P P P P P referenced
- 26. File synchronization syscalls open(.., O_SYNC | O_DSYNC) fsync(int fd) fdatasync(int fd) msync(int *addr, size_t len,..) sync_file_range(int fd, off64_t off, off64_t nbytes,..)
- 27. File synchronization syscalls open(.., O_SYNC | O_DSYNC) fsync(int fd) fdatasync(int fd) msync(int *addr, size_t len,..) sync_file_range(int fd, off64_t off, off64_t nbytes,..) No page subset syncrhonization write(fd, buf, 1GB) – isn't atomic against system failure Some pages can be flushed before synchronization
- 28. Flush out advises posix_fadvise(int fd, off_t offset, off_t len, int advice) ● POSIX_FADV_DONTNEED – invalidate specified pages int invalidate_inode_page(struct page *page) { if (PageDirty(page) || PageWriteback(page)) return 0; madvise(void *addr, size_t length, int advice) MADV_DONTNEED – unmap page table entries, initializes dirty pages flushing
- 29. Lesson 6: Linux VMM as DBMS engine? Linux VMM ● evicts dirty pages ● it doesn't know exactly whether they're still needed (DONTNEED!) ● nobody knows when the pages are synced ● checkpoint is typically full database file sync ● performance: locked scans for free/clean pages by timeouts and no-memory Don't use mmap() if you want consistency!
- 30. Transactional filesystems: Reiser4 Hybrid TM: Journaling or Write-Anywhere (Copy-On-Write) Only small data block writes are transactional Full transaction support for large writes isn't implemented
- 31. Transactional filesystems: BtrFS Uses log-trees, so [probaly] can be used instead of doublewrite buffer ioctl(): BTRFS_IOC_TRANS_START and BTRFS_IOC_TRANS_END
- 32. Transactional filesystems: others Valor R.P.Spillane et al, “Enabling Transactional File Access via Lightweight Kernel Extensions”, FAST'09 ● Transactions: kernel module betweem VFS and filesystem ● New transactional syscalls (log begin, log append, log resolve, transaction sync, locking) ● patched pdflush for eviction in proper order Windows TxF ● deprecated
- 33. Transactional operating systems TxOS D.E.Porter et al., “Operating System Transactions”, SOSP'09 ● Almost any sequence of syscalls can run in transactional context ● New transactional syscalls (sys_xbegin, sys_xend, sys_xabort) ● Alters kernel data structures by transactional headers ● Shadow-copies consistent data structures ● Properly resolves conflicts between transactional and non- transactional calls
- 34. The same OS does the right job Failure-atomic msync() S.Park et al., “Failure-Atomic msync(): A Simple and Efficient Mechanism for Preserving the Integrity of Durable Data”, Eurosys'13. ● No voluntary page writebacks: MAP_ATOMIC for mmap() ● Jounaled writeback – msync() – REDO logging – page writebacks
- 35. Record-oriented filesystem OpenVMS Record Management Service (RMS) ● Record formats: fixed length, variable length, stream ● Access methods: sequential, relative record number, record address, index ● sys$get() & sys$put() instead of read() and write()
- 36. TempestaDB Is part of TempestaFW (a hybrid of firewall and Web-accelerator) In-memory database for Web-cache and firewall rules (must be fast!) Stonebreaker's “The Traditional RDBMS Wisdom is All Wrong” Accessed from kernel space (softirq!) as well as user space Can be concurrently accessed by many processes In-progress development
- 37. Kernel database for Web-accelerator?
- 38. Transport http://natsys-lab.blogspot.ru/2015/03/linux-netlink-mmap-bulk-data-transfer.html Collect query results → copy to some buffer Zero-copy mmap() to user-space Show to user
- 39. TempestaDB internals Preallocates large pool of huge pages at boot time ● so full DB file mmap() is compensated by huge pages ● 2MB extent = huge page Tdbfs is used for custom mmap() and msync() for persistency mmap() => record-orientation out of the box No-steal force or no-force buffer management no need for doublewrite buffer undo and redo logging is up to application Automatic cache eviction
- 41. TempestaDB: trx write (no-steal)
- 46. NUMA replication
- 47. NUMA sharding
- 48. Memory optimized Cache conscious Burst Hash Trie ● short offsets instead of pointers ● (almost) lock-free lock-free block allocator for virtually contiguous memory
- 49. Burst Hash Trie
- 50. Burst Hash Trie
- 51. Burst Hash Trie
- 52. Burst Hash Trie
- 53. Burst Hash Trie: transactions
- 54. Thanks! Availability: https://github.com/tempesta-tech/tempesta Blog: http://natsys-lab.blogspot.com E-mail: ak@tempesta-tech.com We are hiring!