Adventures with the ClickHouse ReplacingMergeTree Engine
1 like612 views
The document discusses the integration of OLTP databases with ClickHouse using the replacingmergetree engine for efficient data mirroring and replication. It explains key differences between storage organization in OLTP and OLAP, emphasizing the role of primary keys and how row updates are handled, including merging strategies and policy applications. Finally, it provides performance tips and links to further resources on ClickHouse documentation and related tools.
Adventures with the ClickHouse ReplacingMergeTree Engine
- 1. © 2022 Altinity, Inc. Adventures with the ClickHouse ReplacingMergeTree Engine Mirroring data from OLTP databases to ClickHouse Robert Hodges & Altinity Engineering 1 14 December 2022
- 2. © 2022 Altinity, Inc. Let’s make some introductions ClickHouse support and services including Altinity.Cloud Authors of Altinity Kubernetes Operator for ClickHouse and other open source projects Robert Hodges Database geek with 30+ years on DBMS systems. Day job: Altinity CEO Altinity Engineering Database geeks with centuries of experience in DBMS and applications 2
- 3. © 2022 Altinity, Inc. How do different database types arise? 3 eCommerce Inventory management and purchasing vs Funnel analysis and fraud detection Digital Marketing Campaign management vs Campaign evaluation Software Defined Networking Network topology and micro-segment definition vs Access patterns analysis MySQL - OLTP ClickHouse - Analytics
- 4. © 2022 Altinity, Inc. OLTP vs OLAP – Key difference is storage organization 4 ClickHouse Read only selected columns Rows minimally or not compressed Columns highly compressed PostgreSQL, MySQL Read all columns in row
- 5. © 2022 Altinity, Inc. 5 Some data just needs a copy in a column store WordPress dynamic site data eCommerce transactions Ad bidding transactions Online auction transactions Chat messages Mobile provisioning requests Credit card transactions Financial market transactions
- 6. © 2022 Altinity, Inc. Mirroring copies data and keeps it up-to-date 6 Initial Dump/Load MySQL ClickHouse OLTP App Analytic App MySQL Binlog Real-time Replication
- 7. © 2022 Altinity, Inc. So…What’s the problem? 7 Mutating Rows
- 8. © 2022 Altinity, Inc. © 2022 Altinity, Inc. Basics of ReplacingMergeTree 8
- 9. © 2022 Altinity, Inc. Let’s consider a source table in MySQL CREATE TABLE `film` ( `film_id` smallint unsigned NOT NULL AUTO_INCREMENT, `title` varchar(128) NOT NULL, . . . `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`film_id`), KEY `idx_title` (`title`), . . . ) ENGINE=InnoDB 9 Primary key
- 10. © 2022 Altinity, Inc. CREATE TABLE sakila.film ( `film_id` UInt16, `title` String, `description` Nullable(String), `release_year` Nullable(String), . . . `last_update` DateTime, `_version` UInt64 DEFAULT 0, `_sign` Int8 DEFAULT 1 ) ENGINE = ReplacingMergeTree(_version) ORDER BY language_id, studio_id, film_id Create a ClickHouse table to contain the mirrored data 10
- 11. © 2022 Altinity, Inc. How ReplacingMergeTree works 11 0 3 3 1001 . . . . . . 1001 . . . . . . 1001 INSERT _version +1 -1 +1 _sign (Other data columns) fiilm _id UPDATE DELETE language_id 5 1001 . . . . . . -1 studio_id Eventually consistent replacement of rows
- 12. © 2022 Altinity, Inc. INSERT INTO sakila.film VALUES (1001,'Blade Runner','Best. Sci-fi. Film. Ever.', '1982',1,NULL,6,'0.99',117,'20.99','PG', 'Deleted Scenes,Behind the Scenes',now() ,0,1) SELECT title, release_year FROM film WHERE film_id = 1001 ┌─title────────┬─release_year─┐ │ Blade Runner │ 1982 │ └──────────────┴──────────────┘ Adding a row to RMT table 12
- 13. © 2022 Altinity, Inc. INSERT INTO sakila.film VALUES (1001,'Blade Runner', 'Best. Sci-fi. Film. Ever.',...,3,-1), (1001,'Blade Runner - Director''s Cut','Best. Sci-fi. Film. Ever.',...,3,1) SELECT title, release_year FROM film WHERE film_id = 1001 ┌─title─────────────────────────┬─release_year─┐ │ Blade Runner - Director's Cut │ 1982 │ └───────────────────────────────┴──────────────┘ ┌─title────────┬─release_year─┐ │ Blade Runner │ 1982 │ └──────────────┴──────────────┘ Updating a row in the RMT table 13 Unmerged rows!
- 14. © 2022 Altinity, Inc. Rows are replaced when merges occur 14 0 3 3 1001 1 . . . 1001 1 . . . 1001 2 +1 -1 +1 Part Part Merged Part 3 1001 1 -1 3 1001 2 +1 X Pro tip: never assume rows will merge full ?
- 15. © 2022 Altinity, Inc. SELECT film_id, title FROM sakila.film FINAL WHERE film_id = 1001 ┌─title─────────────────────────┬─release_year─┐ │ Blade Runner - Director's Cut │ 1982 │ └───────────────────────────────┴──────────────┘ FINAL keyword merges data dynamically 15 Adds initial scan to merge rows
- 16. © 2022 Altinity, Inc. INSERT INTO sakila.film VALUES (1001,'Blade Runner - Director''s Cut', 'Best. Sci-fi. Film. Ever.',...,5,-1) SELECT title, release_year, _version, _sign FROM sakila.film FINAL WHERE film_id = 1001 ┌─title─────────────────────────┬─release_year─┬─_version─┬─_sign─┐ │ Blade Runner - Director's Cut │ 1982 │ 5 │ -1 │ └───────────────────────────────┴──────────────┴──────────┴───────┘ Deleting a row in RMT table 16 Deleted row!
- 17. © 2022 Altinity, Inc. CREATE ROW POLICY sakila_film_rp ON sakila.film FOR SELECT USING sign != -1 TO ALL SELECT title, release_year, _version, _sign FROM sakila.film FINAL WHERE film_id = 1001 Ok. 0 rows in set. Elapsed: 0.005 sec. Row policies prevent deleted rows from showing up 17 Predicate automatically added to queries
- 18. © 2022 Altinity, Inc. SELECT inventory_id, film_id, title FROM sakila.inventory AS i INNER JOIN sakila.film AS f ON i.film_id = f.film_id WHERE film.film_id = 1001 ┌─inventory_id─┬─film_id─┬─title─────────────────────────┐ │ 1 │ 1001 │ Blade Runner - Director's Cut │ │ 1 │ 1001 │ Blade Runner │ └──────────────┴─────────┴───────────────────────────────┘ JOINs are tricky with RMT 18 Right side table does not have FINAL!
- 19. © 2022 Altinity, Inc. SELECT inventory_id, f.film_id, title FROM sakila.inventory AS i INNER JOIN ( SELECT film_id, title FROM sakila.film FINAL ) AS f ON i.film_id = f.film_id WHERE f.film_id = 1001 ┌─inventory_id─┬─film_id─┬─title─────────────────────────┐ │ 1 │ 1001 │ Blade Runner - Director's Cut │ └──────────────┴─────────┴───────────────────────────────┘ Use a subquery with FINAL for right hand side table 19 Not elegant, but we can work with it
- 20. © 2022 Altinity, Inc. © 2022 Altinity, Inc. Performance tips 20
- 21. © 2022 Altinity, Inc. CREATE TABLE sakila.film ( `film_id` UInt16, `title` String, . . . `_version` UInt64 DEFAULT 0, `_sign` Int8 DEFAULT 1 ) ENGINE = ReplacingMergeTree(_version) ORDER BY language_id, studio_id, film_id ORDER BY is critical for MergeTree performance 21 Row key goes on right Other cols go to left Pro tip: Use PRIMARY KEY to prefix a long ORDER BY
- 22. © 2022 Altinity, Inc. Use care on updates when ORDER BY has > 1 column INSERT INTO sakila.film VALUES (1001,'Blade Runner','Best. Sci-fi. Film. Ever.', '1982',1,NULL,6,'0.99',117,'20.99','PG', 'Deleted Scenes,Behind the Scenes',now() ,3,-1), (1001,'Blade Runner - Director''s Cut', 'Best. Sci-fi. Film. Ever.', '1982',2,NULL,6,'0.99',120,'20.99','PG', 'Deleted Scenes,Behind the Scenes',now() ,3,1) 22 Must delete row if ORDER BY columns change!
- 23. © 2022 Altinity, Inc. CREATE TABLE sakila.film ( `film_id` UInt16, `title` String, . . . `_version` UInt64 DEFAULT 0, `_sign` Int8 DEFAULT 1 ) ENGINE = ReplacingMergeTree(_version) PARTITION BY intDiv(film_id, 10000000) ORDER BY language_id, studio_id, film_id Partitioning is important for large tables 23 Choose a partition key that keeps row changes local to single partitions
- 24. © 2022 Altinity, Inc. SELECT release_year, count() FROM sakila.film FINAL GROUP BY release_year ORDER BY release_year SETTINGS do_not_merge_across_partitions_select_final = 1 Restrict FINAL merge scope to single partitions 24 Run faster; parallelize across partitions Note: Run ClickHouse 22.10+ to use this feature* * https://github.com/ClickHouse/ClickHouse/issues/43296
- 25. © 2022 Altinity, Inc. © 2022 Altinity, Inc. Current work to improve ReplacingMergeTree 25
- 26. © 2022 Altinity, Inc. Just update from my side: the param min_age_to_force_merge_on_partition_only works Alexandr DubovikovSETTINGS min_age_to_force_merge_seconds = 120, min_age_to_force_merge_on_partition_only = true; Alexandr Dubovikovit will merge all parts at once 26
- 27. © 2022 Altinity, Inc. # https://github.com/ClickHouse/ClickHouse/pull/40945 SET force_select_final = 1 SELECT inventory_id, film_id, title FROM sakila.inventory AS i INNER JOIN sakila.film AS f ON i.film_id = f.film_id WHERE film.film_id = 1001 ┌─inventory_id─┬─film_id─┬─title─────────────────────────┐ │ 1 │ 1001 │ Blade Runner - Director's Cut │ └──────────────┴─────────┴───────────────────────────────┘ Add implicit FINAL at query level (Altinity) 27 Add FINAL automatically to table
- 28. © 2022 Altinity, Inc. # https://github.com/ClickHouse/ClickHouse/pull/41005 CREATE TABLE sakila.film ( `film_id` UInt16, . . . `_version` UInt64 DEFAULT 0, `_sign` UInt8 DEFAULT 1 ) ENGINE = ReplacingMergeTree(_version, _sign) ORDER BY language_id, studio_id, film_id Eliminate deleted rows automatically (ContentSquare) 28 Delete column processed by RMT engine
- 29. © 2022 Altinity, Inc. © 2022 Altinity, Inc. Wrap up 29
- 30. © 2022 Altinity, Inc. Fully wired, continuous replication based on RMT 30 Table Engine(s) Initial Dump/Load MySQL ClickHouse OLTP App Analytic App MySQL Binlog Debezium Altinity Sink Connector Kafka* Event Stream *Including Pulsar and RedPanda ReplacingMergeTree
- 31. © 2022 Altinity, Inc. Where is the documentation? ClickHouse official docs – https://clickhouse.com/docs/ Altinity Blog – https://altinity.com/blog/ Altinity Sink Connector for ClickHouse – https://github.com/Altinity/clickhouse-sink-connector Altinity Knowledge Base – https://kb.altinity.com/ 31
- 32. © 2022 Altinity, Inc. Thank you! Questions? rhodges at altinity dot com https://altinity.com 32