MyRocks in MariaDB: why and how
0 likes1,343 views
This document discusses porting MyRocks, a storage engine that combines RocksDB with MySQL, to MariaDB. It provides an overview of MyRocks, the tasks involved in porting it to MariaDB, the current status, and future plans. Key points include porting MyRocks from Facebook's MySQL to MariaDB, building packages, releasing it as part of a MariaDB version, addressing failing tests and missing features, and improving integration with MariaDB capabilities like binlogging. The goal is to get MyRocks adopted more broadly by adding it to MariaDB and expanding the community around it.
MyRocks in MariaDB: why and how
- 1. Sergei Petrunia sergey@mariadb.com MyRocks in MariaDB Why and How MariaDB Developer UnConference April 9-10th 2017 New York
- 2. 11:00:56 2 About MariaDB What is MyRocks?
- 3. 11:00:56 3 What is MyRocks ● RocksDB + MySQL = MyRocks ● LSM architecture ● Better compression ● Better IO efficiency ● Write optimizations − Bulk load − No-read writes ● Used and verified at Facebook
- 4. 11:00:56 4 MyRocks needs adoption and community ● Currently, it is in github.com/facebook/mysql5.6 ● No binaries ● No packages ● Very much “in-house” experience − Special branch of MySQL − Special way to compile − Special command to run tests − ...
- 5. 11:00:56 5 MyRocks + MariaDB = ♥ ● New technology ● Adoption ● Packaging ● Community ● MariaDB features
- 6. 11:00:56 6 Plan 1. Tasks for porting MyRocks 2. Current status 3. Future
- 7. 11:00:56 7 Tasks for putting MyRocks into MariaDB ● Port MyRocks from facebook/mysql-5.6 to MariaDB ● Do builds and packages ● Catch a MariaDB release train
- 8. 11:00:56 8 Checklist for putting MyRocks into MariaDB ● Port MyRocks from facebook/mysql-5.6 to MariaDB ● Do builds and packages ● Catch a MariaDB release train
- 9. 11:00:56 9 Porting MyRocks SQL layer RocksDB MyRocks
- 10. 11:00:56 10 Porting MyRocks SQL layer RocksDB Read-free replication More counters Gap lock use detector Extra range info Bloom filter use MyRocks ... counter++ ... ● MyRocks is not just storage/rocksdb/*
- 11. 11:00:56 11 Porting MyRocks ● Resolve the SQL layer differences ● Resolve MariaDB vs MySQL differences − Index Condition pushdown − Extended Keys − Group Commit − EXPLAIN output changes − Extra/missing warnings − …
- 12. 11:00:56 12 Checklist for putting MyRocks into MariaDB ● Port MyRocks from facebook/mysql-5.6 to MariaDB ● Do builds and packages ● Catch a MariaDB release train
- 13. 11:00:56 13 Do builds and packages ● Facebook/mysql-5.6 builds statically ● We want ha_rocksdb.so ● We want packages − deb, rpm, bintar, windows zip, ... ● RocksDB uses compression libraries − Need to link against them − Need to have proper dependencies
- 14. 11:00:56 14 Checklist for putting MyRocks into MariaDB ● Port MyRocks from facebook/mysql-5.6 to MariaDB ● Do builds and packages ● Catch a MariaDB release train
- 15. 11:01:01 15 MariaDB releases ● MariaDB 10.0: − Alpha: 12 Nov 2012 − Stable: 31 Mar 2014 ● MariaDB 10.1 (Stable) − Alpha: 30 Jun 2014 − Stable: 17 Oct 2015 ● MariaDB 10.2 (RC) − Alpha: 18 Apr 2016 − RC: 17 Feb 2017 − Stable: soon Get into a MariaDB Release before it is declared stable ● MariaDB 10.3 − “alpha soon”
- 16. 11:01:01 16 Plugin Maturity ● Plugins declare their maturity − unknown, experimental, alpha, beta, gamma, stable ● mysqld --plugin-maturity=level − will not load plugins less mature than level. ● This is how MyRocks can get into MariaDB 10.2 − Plugin with maturity=alpha.
- 18. 11:01:01 18 Current status ● MariaDB 10.2.5 RC2 includes MyRocks ● Features ● Packaging ● Quality
- 19. 11:01:01 19 Current status ● MariaDB 10.2.5 RC2 includes MyRocks ● Features ● Packaging ● Quality
- 20. 11:00:56 20 Feature status ● Working − General server functionality − Optimizer-related ● Bloom filter support − Tests ● Non-working yet − Interplay between the storage engine and the binlog ● Group commit ● Read-Free-Slave ● GTID − Gap Lock detector
- 21. 11:01:01 21 diff -u mariarocks myrocks 3K line diff for the code - CMake build changes - MariaDB's differences in Extended Keys - MariaDB's differences in Index Condition Pushdown - MariaDB calls ha_statistic_increment(...) above the storage engine - Implement prepare_*_scan(), Bloom Filter works for ORDER BY DESC - Storage Engine API function signature changes - s/MySQL/MariaDB/ in names of structs, constants, etc - #ifdef-away code related to replication and binlog - #ifdef-away extra diagnostics like SHOW ENGINE TRANSACTION STATUS - ... 8K lines diff for tests: - MariaDB has different default values for SQL columns - EXPLAIN output is slightly different - Handler_xxx counters counted a bit differently - MTR explicitly logs connection open/switch/etc - mysql-test/suite/$MYROCKS_TEST → storage/rocksdb/mysql-test/$MYROCKS_TEST - ...
- 22. 11:00:56 22 Gap Lock Detector ● SQL level feature ● Detects queries that use gap locking ● Workflow − Run with InnoDB, detect and remove the querie − Migrate to MyRocks ● Objections to pushing − MyRocks has features on SQL layer! ● ?
- 23. 11:01:01 23 Current status ● MariaDB 10.2.5 RC2 includes MyRocks ● Features ● Packaging ● Quality
- 24. 11:01:01 24 Packaging in facebook/mysql-5.6 ● MyRocks is linked statically ● RocksDB is a submodule (@revno) − Not a dependency ● Can compile with compression libraries − Snappy − Zlib − ZStandard − ...
- 25. 11:01:01 25 Packaging in MariaDB ● MyRocks is built as ha_rocksdb.so ● RocksDB is a submodule (@revno) − Linked in statically in the .so ● Compression libraries − Bintar: links statically (snappy, zlib) − .deb/.rpm: package has a dependency (snappy, zlib) − Windows zip/MSI: zlib ● rocksdb_hotbackup is missing − but works (have an MDEV for this)
- 26. 11:01:01 26 Compiling in MariaDB ● MyRocks is built as ha_rocksdb.so ● MyRocks is compiled by default − Except for unsupported platforms ● MyRocks tests can be run as regular tests ./mysql-test-run rocksdb.type_varchar
- 27. 11:01:01 27 Current status ● MariaDB 10.2.5 RC2 includes MyRocks ● Features ● Packaging ● Quality
- 28. 11:01:01 28 MyRocks Quality ● Alpha-maturity plugin in RC-level MariaDB Server ● MyRocks tests are run in MariaDB buildbot ● There are failing testcases − Not necessarily MyRocks bugs − Working on this ● There are missing features − Can finish those that touch only storage/rocksdb − May be harder if the feature touches the SQL layer.
- 30. 11:01:01 30 Future ● Fix the failing tests − Some failures are from the upstream ● Usability improvements ● Interplay between the Storage Engine and Binlog − Get it to work − MariaDB's Group Commit − Per-engine mysql.slave_gtid_pos − Parallel Slave support? ● Keep merging from upstream.