In-memory OLTP storage with persistence and transaction support
3 likes1,952 views
The document discusses the development of a prototype for in-memory OLTP storage in PostgreSQL that supports persistence and transaction management. It explores the motivations for pluggable storage system capabilities, potential use cases, and the current state of the implementation. The prototype uses a foreign data wrapper (FDW) and employs a unique in-memory engine design that differs from traditional methods, though it currently has limitations and is not yet production-ready.
In-memory OLTP storage with persistence and transaction support
- 1. In-memory OLTP storage with persistence and transac on support Alexander Korotkov Postgres Professional October 25, 2017 Alexander Korotkov In-memory OLTP storage with persistence and transac on support 1 / 34
- 2. Disclaimer ▶ This talk is not about something produc on ready. Don’t hold your breath while wai ng for use some of considered func onality in produc on. When this func onality will be available and produc on-ready, it might become something drama cally different. ▶ This talk is about some intermediate results achieved during development. These results are presented for discussion and brainstorming in order to make further development be er. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 2 / 34
- 3. What this talk is about? ▶ We (Postgres Pro) have a prototype of in-memory OLTP storage implemented using FDW. ▶ It’s proof of concept of opportuni es for in-memory OLTP in PostgreSQL (it was debatable that there are any). ▶ It’s yet another example of alterna ve storage implemented using FDW interface before we’ve na ve pluggable storages. So, it’s waypoint to verify where we are on pluggable storages. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 3 / 34
- 4. Why pluggable storages? ▶ Lack of pluggable storages support is understood as limita on. PostgreSQL was always posi oned as highly extendable DBMS while lack of pluggable storages support is large gap in this area. ▶ Rising interest in PostgreSQL from enterprises. Postgres-centric companies have enough of resources to support mul ple storage engines. Enterprises are also interested in using PostgreSQL for non-OLTP tasks. Alterna ve storages might improve OLTP too (UNDO log for be er update performance). Alexander Korotkov In-memory OLTP storage with persistence and transac on support 4 / 34
- 5. Use cases for pluggable storages ▶ Different MVCC implementa on: mostly varia ons of UNDO log, but not only. ▶ Data compression: either row-level, page-level etc... ▶ Non disk-oriented storage: in-memory, SSD-op mized, NVRAM-op mized. ▶ Non heap-like rows layout: index-organized table (IOT) including LSM. ▶ Non row-oriented data layout: either column or parquet layouts. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 5 / 34
- 6. Current state of pluggable storage https://www.postgresql.org/message-id/flat/ 20160812231527.GA690404%40alvherre.pgsql ▶ Started as quite mechanical separa on of heap_* methods into storage AM interface. ▶ Boundary of storage layer was significantly shi ed during discussions. ▶ S ll a lot of work to do. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 6 / 34
- 7. Current view on pluggable storage proper es ▶ All the storages should share same transac on model (or have no transac ons at all). ▶ All the storages should write same WAL stream. ▶ Tuples have to be iden fied by TIDs (further improvement is possible). ▶ Storages should share some of index access methods. ▶ Index access method interface should be expanded with new func ons (at least retail tuple delete). ▶ Storages may have completely different MVCC implementa on. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 7 / 34
- 8. Why FDW for prototyping Pro: ▶ FDW is completely free in the way it scans and modifies the foreign table. ▶ This approach is already used in cstore_fdw1 , vops2 . Cons: ▶ Lack of control on associated resources, ▶ Lack of DDL support. 1 https://github.com/citusdata/cstore_fdw 2 https://github.com/postgrespro/vops Alexander Korotkov In-memory OLTP storage with persistence and transac on support 8 / 34
- 9. Why in-memory? ▶ No extra mapping layer (buffer manager) to traverse from one page to another. ▶ Row-level WAL takes less space (no page-level informa on, no explicit index logging), but slower to apply. ▶ Be er IO u liza on (write both snapshots and WAL are wri en sequen ally). Alexander Korotkov In-memory OLTP storage with persistence and transac on support 9 / 34
- 10. What this par cular in-memory engine is? ▶ Index organized table where index is in-memory B-tree. ▶ This B-tree supports transac ons and MVCC using UNDO log which is circular buffer in memory containing both row-level and page-level records. ▶ It writes full data snapshots on checkpoints and row-level WAL. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 10 / 34
- 11. Why our in-memory engine is a good example of pluggable storage Because it does the things in a quite different way. ▶ It stores data in main memory with quite different model of persistence: full data snapshots plus row-level WAL. ▶ It doesn’t have heap-like layout. ▶ It uses very different MVCC implementa on: combina on of row-level and page-level undo logs. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 11 / 34
- 12. Why our in-memory engine is a bad example of pluggable storage ▶ It uses CSN snapshot model which is far from ge ng commi ed. ▶ Tuples aren’t iden fied by TIDs. ▶ Persistence is implemented using set of hacks. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 12 / 34
- 13. Configura on parameters ▶ in_memory_engine.shared_pool_size – size of separate pool of 1k pages for in-memory tables. ▶ in_memory_engine.undo_size – size of circular buffer for undo records to support transac ons and MVCC. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 13 / 34
- 14. Usage: defining in-memory table and inser ng data CREATE EXTENSION in_memory; CREATE FOREIGN TABLE im_test ( id int8 NOT NULL, val text NOT NULL ) SERVER in_memory OPTIONS (indices ’unique (id)’, persistent ’true’); INSERT INTO im_test (SELECT id, ’val’ || id FROM generate_series(1, 1000000) id); Alexander Korotkov In-memory OLTP storage with persistence and transac on support 14 / 34
- 15. Usage: querying a single key # EXPLAIN ANALYZE SELECT * FROM im_test WHERE id = 50000; QUERY PLAN ---------------------------------------------------------------- Foreign Scan on im_test (cost=0.06..4.52 rows=357 width=40) (actual time=0.190..0.191 rows=1 loops=1) Pk conds: (id = 50000) Planning time: 0.635 ms Execution time: 0.260 ms (4 rows) Alexander Korotkov In-memory OLTP storage with persistence and transac on support 15 / 34
- 16. Usage: querying a key range # EXPLAIN ANALYZE SELECT * FROM im_test WHERE id >= 10000 AND id <= 20000; QUERY PLAN ---------------------------------------------------------------- Foreign Scan on im_test (cost=0.06..5.41 rows=357 width=40) (actual time=0.045..4.194 rows=10001 loops=1) Pk conds: (id >= 10000 AND id <= 20000) Planning time: 0.075 ms Execution time: 4.915 ms (4 rows) Alexander Korotkov In-memory OLTP storage with persistence and transac on support 16 / 34
- 17. Usage: querying for non-key condi on # EXPLAIN ANALYZE SELECT * FROM im_test WHERE val LIKE ’%1111%’; QUERY PLAN ---------------------------------------------------------------- Foreign Scan on im_test (cost=0.06..891.83 rows=571 width=40) (actual time=0.345..187.325 rows=280 loops=1) Filter: (val ~~ ’%1111%’::text) Rows Removed by Filter: 999720 Planning time: 0.046 ms Execution time: 187.375 ms (5 rows) Alexander Korotkov In-memory OLTP storage with persistence and transac on support 17 / 34
- 18. Usage: non-persistent tables are writable on standby *** Master *** # CREATE FOREIGN TABLE im_test (id int8 NOT NULL, val text NOT NULL) SERVER in_memory OPTIONS (indices ’unique (id)’, persistent ’false’); # INSERT INTO im_test (SELECT id, ’val’ || id FROM generate_series(1, 1000000) id); INSERT 0 1000000 *** Standby *** # SELECT * FROM im_test; id | val ----+----- (0 rows) # INSERT INTO im_test VALUES (1, ’foo’), (2, ’bar’); INSERT 0 2 # SELECT * FROM im_test; id | val ----+----- 1 | foo 2 | bar (2 rows) Alexander Korotkov In-memory OLTP storage with persistence and transac on support 18 / 34
- 19. Limita ons ▶ Only B-tree with limited func onality is supported. ▶ No secondary indexes are supported yet. ▶ No out-of-line storage are supported for tuples yet. ▶ Undo log shouldn’t wraparound during single transac on (that transac on is automa cally aborted). ▶ If required undo record is already overflowed then “snapshot’s too old” error is emi ed. ▶ Serializable isola on level isn’t supported. ▶ Replica on isn’t supported yet. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 19 / 34
- 20. Read-only benchmark 0 50 100 150 200 250 # Clients 0 200000 400000 600000 800000 1000000 1200000 1400000 1600000 QPS pgbench -s 1000 -j $n -c $n -M prepared -S on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 builtin in-memory Alexander Korotkov In-memory OLTP storage with persistence and transac on support 20 / 34
- 21. Why there is no win? Storage is only one layer par cipa ng in query execu on. There are also: ▶ Network layer, ▶ Executor, ▶ Parser (analyze & rewrite if not prepared), ▶ Transac on management (including snapshot acquirement), ▶ ... Alexander Korotkov In-memory OLTP storage with persistence and transac on support 21 / 34
- 22. Measuring overheads 0 50 100 150 200 250 # Clients 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 QPS pgbench -s 1000 -j $n -c $n -M prepared -S on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 read-only SELECT 1; ; Alexander Korotkov In-memory OLTP storage with persistence and transac on support 22 / 34
- 23. Read-only benchmark: fetch 9 values per single query set aid1 random(1, 100000 * :scale) set aid2 random(1, 100000 * :scale) set aid3 random(1, 100000 * :scale) set aid4 random(1, 100000 * :scale) set aid5 random(1, 100000 * :scale) set aid6 random(1, 100000 * :scale) set aid7 random(1, 100000 * :scale) set aid8 random(1, 100000 * :scale) set aid9 random(1, 100000 * :scale) SELECT abalance FROM pgbench_accounts WHERE aid IN (:aid1, :aid2, :aid3, :aid4, :aid5, :aid6, :aid7, :aid8, :aid9); Alexander Korotkov In-memory OLTP storage with persistence and transac on support 23 / 34
- 24. Read-only benchmark: fetch 9 values per single query 0 50 100 150 200 250 # Clients 0 200000 400000 600000 800000 1000000 1200000 QPS pgbench -s 1000 -j $n -c $n -M prepared -f ro9.sql on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 builtin in-memory Alexander Korotkov In-memory OLTP storage with persistence and transac on support 24 / 34
- 25. Read-only benchmark: compare values-per-second 0 50 100 150 200 250 # Clients 0 2000000 4000000 6000000 8000000 10000000 VPS pgbench -s 1000 -j $n -c $n -M prepared -S on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 builtin-1 builtin-9 in_memory-1 in_memory-9 Alexander Korotkov In-memory OLTP storage with persistence and transac on support 25 / 34
- 26. Read-write benchmark without persistence (async commit) 0 50 100 150 200 250 # Clients 0 50000 100000 150000 200000 250000 TPS pgbench -s 1000 -j $n -c $n -M prepared on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 unlogged table in_memory Alexander Korotkov In-memory OLTP storage with persistence and transac on support 26 / 34
- 27. Read-write benchmark with persistence (async commit) 0 50 100 150 200 250 # Clients 0 50000 100000 150000 200000 TPS pgbench -s 1000 -j $n -c $n -M prepared on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 builtin in-memory Alexander Korotkov In-memory OLTP storage with persistence and transac on support 27 / 34
- 28. Read-write benchmark: do transac on in a single statement CREATE OR REPLACE FUNCTION tcpb_trx(_aid int, _bid int, _tid int, _delta int) RETURNS void AS $$ BEGIN UPDATE pgbench_accounts SET abalance = abalance + _delta WHERE aid = _aid; PERFORM abalance FROM pgbench_accounts WHERE aid = _aid; UPDATE pgbench_tellers SET tbalance = tbalance + _delta WHERE tid = _tid; UPDATE pgbench_branches SET bbalance = bbalance + _delta WHERE bid = _bid; INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (_tid, _bid, _aid, _delta, CURRENT_TIMESTAMP); END; $$ LANGUAGE plpgsql; set aid random(1, 100000 * :scale) set bid random(1, 1 * :scale) set tid random(1, 10 * :scale) set delta random(-5000, 5000) SELECT tcpb_trx(:aid, :bid, :tid, :delta); Alexander Korotkov In-memory OLTP storage with persistence and transac on support 28 / 34
- 29. Read-write benchmark with persistence (async commit, func on vs. interac ve) 0 50 100 150 200 250 # Clients 0 100000 200000 300000 400000 500000 600000 TPS pgbench -s 1000 -j $n -c $n -M prepared on 4 x 18 cores Intel Xeon E7-8890 processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = 300 builtin builtin-func in_memory in_memory-func Alexander Korotkov In-memory OLTP storage with persistence and transac on support 29 / 34
- 30. Hacks used in implementa on ▶ Minimalis c CSN implementa on CSNs are assigned but neither used, neither wri en to SLRUs. in-memory engine doesn’t need SLRU. ▶ Checkpoint hook in-memory engine writes full data snapshot on checkpoint. ▶ Generic logical message hook Used to implement custom recovery/replica on. This is an awful hack. ▶ TRUNCATE using u lity command hook TRUNCATE isn’t supported by FDW directly. ▶ DROP support using event trigger Used to free the resources occupied by in-memory table. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 30 / 34
- 31. Recovery problem ▶ Row-level WAL is compact, but it requires meta-informa on to apply. That is we need to be able to read system catalog while applying WAL including recovery. ▶ We can’t access system catalog during recovery, because the whole database isn’t accessible since it’s not recovered yet. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 31 / 34
- 32. Recovery problem solu on: 2-phase recovery At the second phase we have consistent system catalog. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 32 / 34
- 33. Future roadmap Integrate in-memory as pluggable storage: ▶ In-memory B-tree as index access method. ▶ Implement storage for in-memory tables using one of following ways: ▶ Write some kind of «in-memory heap» OR/AND ▶ Write a storage wrapper implemen ng index-organized table. Alexander Korotkov In-memory OLTP storage with persistence and transac on support 33 / 34
- 34. Thank you for a en on! Alexander Korotkov In-memory OLTP storage with persistence and transac on support 34 / 34