Highload2o013 osipv
0 likes390 views
The document discusses several popular algorithms for storing data on disk, including B-trees, LSM-trees, and cache-oblivious B-trees. B-trees balance speed of insertion, update, and selection but can be slow for deletes. LSM-trees are suitable for data with different degrees of recency and allow segregating data between memory and disk storage. Cache-oblivious data structures achieve optimal performance regardless of cache parameters.
Highload2o013 osipv
- 1. Популярные алгоритмы хранения данных на диске Konstantin Osipov, kostja@tarantool.org October 28th, 2013
- 2. Случай в квадрате 36-80 • • • B-tree – most popular disk-based data structure B-tree balances INSERT, UPDATE and SELECT speed DELETEs can be slow
- 3. СУБД быстрая, настраивать надо уметь
- 6. Что означает сache-oblivious? (2) BLOCKMULT(A,B,C,n): 1 for i = 1 to n/s do: 2 for j = 1 to n/s do: 3 for k = 1 to n/s do: 4 ORDMULT(Aik, Bkj, Cij, s)
- 11. LSM-tree: применение ● Данные с разной степенью актуальности – – Стена в соцсети – Чаты – ● Ленты сообщений События Сегрегация данных – Данные в LSM, индекс в памяти
- 12. COLA: архитектура O(logB(N)) vs. O(logB(N)/B)
- 14. 16 37 Self-Balancing Tree Memory Disk WAL: 37, 16
- 15. 7 41 Self-Balancing Tree Memory 16 37 Sorted String Table WAL: 41, 7, 37, 16 Disk
- 16. Memory 7 37 7 16 37 41 WAL: 41, 7, 28, 16 Disk
- 17. 10 28 Memory 7 37 Disk 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16
- 18. Memory 10 28 7 16 37 41 WAL: 10, 28, 41, 7, 37, 16 Disk
- 19. 2 47 Memory 10 28 Disk 7 16 37 41 WAL: 47, 2, 10, 28, 41, 7, 37, 16
- 20. 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
- 21. 6 49 2 28 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
- 22. 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
- 23. 23 32 6 49 2 10 28 41 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
- 24. 6 32 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16 Memory Disk
- 25. 30 45 Memory 6 32 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 26. 14 38 Memory 30 45 Disk 6 23 32 49 2 7 10 16 28 37 41 47 WAL: 38, 14, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 27. 6 10 Memory 2 30 Disk 2 14 30 41 2 2 6 7 14 23 30 37 41 47 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 28. Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 29. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 30. GET(16) Memory 22 37 Disk 10 25 36 42 3 2 6 8 15 26 35 40 45 48 7 10 14 16 23 28 30 32 37 38 41 45 47 49 WAL: 37, 22, 36, 10, 25, 42, 10, 6, 38, 14, 45, 30, 45, 32, 23, 49, 6, 47, 2, 10, 28, 41, 7, 37, 16
- 33. Sophia: архитектура Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
- 34. 39, 16, 85, 96 Insert Memory Key-value index Disk WAL Page index
- 36. 31, 25, 10, 86 Insert 16, 96 Memory Key-value index Disk Page index WAL 16, 39, 85, 96
- 37. Memory 10 25 31 86 Key-value index 16, 96 Page index merge Disk 16, 39, 85, 96
- 38. Memory 10 25 31 86 Key-value index 10, 31 39, 96 Page index split Disk 10, 16, 25, 31 39, 85, 86, 96
- 39. 15, 26, 40, 84 Insert 10, 31 Memory Key-value index Disk WAL 39, 96 Page index 10, 16, 25, 31 39, 85, 86, 96
- 40. Memory 15 26 40 84 Key-value index 10, 31 39, 96 Page index merge Disk 10, 16, 25, 31 39, 85, 86, 96
- 41. Memory 15 26 40 84 Key-value index 10, 25 26, 31 39, 85 86, 96 Page index split Disk 10, 15, 16, 25 26, 31 39, 40, 84, 85 86, 96
- 42. Эпилог: choose your db wisely ?
- 43. Links ● ● ● ● ● ● ● ● ● ● Bitcask A Log-Structured Hash Table for Fast Key/Value Data, Justin Sheehy David Smith with inspiration from Eric Brewer The Log-Structured Merge-Tree (LSM-Tree) Patrick O'Neil , Edward Cheng, Dieter Gawlick, Elizabeth O'Neil Cache-Oblivious Algorithms by Harald Prokop (Master theses) Space/time trade-offs in hash coding with allowable errors, Burton H. Bloom Data Structures and Algorithms for Big Databases, Michael A. Bender Stony Brook & Tokutek Bradley C. Kuszmaul (XLDB tutorial) http://github.com/pmwkaa/sophia, http://sphia.org http://codecapsule.com/2012/12/30/implementing-a-key-value-store-part-3-comparative-analysis-of-the-ar http://stackoverflow.com/questions/6079890/cache-oblivious-lookahead-array http://www.youtube.com/watch?v=88NaRUdoWZM(Tim Callaghan: Fractal Tree indexes) http://code.google.com/p/leveldb/downloads/list