How to migrate_to_sharding_with_spider
- 1. How to migrate to sharding with Spider Kentoku SHIBA
- 2. 1. What is SPIDER? 2. Why SPIDER? what SPIDER can do for you? 3. How to migrate to sharding using Replication 4. How to migrate to sharding using Trigger 5. How to migrate to sharding using Spider function 6. How to migrate to sharding using Vertical Partitioning Storage Engine Agenda
- 4. What is the Spider Storage Engine? Spider is a sharding solution and proxying solution. Spider Storage Engine is a plugin of MariaDB/MySQL. Spider tables can be used to federate from other servers MariaDB/MySQL/OracleDB tables as if they stand on local server. And Spider can create database sharding by using table partitioning feature.
- 5. What is the Spider Storage Engine? 1.request 2. Execute SQL 4.response AP All databases can be used as ONE database through Spider. APAP AP AP SPIDER （MariaDB/MySQL） MariaDB tbl_a MySQL tbl_b SPIDER （MariaDB/MySQL） SPIDER （MariaDB/MySQL） OracleDB tbl_c 3. Distributed SQL3. Distributed SQL 3. Distributed SQL
- 6. What is the Spider Storage Engine? Spider is bundled in MariaDB from 10.0 and all patches for MariaDB is applied in 10.3
- 7. Why SPIDER? What SPIDER can do for you?
- 8. Why Spider? What Spider can do for you? For federation You can attach tables from other servers or from local server by using Spider. For sharding You can divide huge tables and huge traffics to multiple servers by using Spider.
- 9. Why Spider? What Spider can do for you? Cross shard join You can join all tables by using Spider, even if tables are on different servers.
- 10. simple sharding solution Join operation with simple sharding solution (without Spider) DB1 tbl_a1 1.Request 2. Execute SQL with JOIN 3.Response DB2 AP Join operation requires that all joined tables are on same server. APAP AP AP tbl_a2tbl_b1 tbl_b2
- 11. Join operation with Spider 1.request 2. Execute SQL with JOIN 3.response AP You can JOIN all tables, even if tables are on different servers. APAP AP AP SPIDER （MariaDB/MySQL） DB1 tbl_a1 DB2 tbl_a2tbl_b1 tbl_b2
- 12. Why Spider? What Spider can do for you? Join push down If it is possible, Spider executes JOIN operation at data node directly.
- 13. JOIN push down 1.request 2. Execute SQL with JOIN 3.response AP If all tables are on same data node, Spider executes JOIN operation on data node directly. APAP AP AP SPIDER （MariaDB/MySQL） DB1 tbl_a DB2 tbl_ctbl_b tbl_d
- 14. JOIN push down Simple join operation are two times faster on simple JOIN pushdown test. Also, in this pushdown of JOIN, when aggregate functions are included in the query, since the aggregation processing is also executed at the data node, the amount of data transfer is greatly reduced and it becomes super high speed.
- 15. How to migrate to sharding with Spider using Replication
- 16. Initial Structure There is 1 MariaDB server without Spider. DB1 tbl_a Create table tbl_a ( col_a int, col_b int, primary key(col_a) ) engine = InnoDB;
- 17. Step 1 (for migrating) Create table on DB3 and DB4. Then create Spider table on DB2. DB1 tbl_a DB3 tbl_a col_a%2=1col_a%2=0 DB2 DB4 tbl_a Create table tbl_a ( col_a int, col_b int, primary key(col_a) ) engine = Spider Connection ‘ table “tbl_a”, user “user”, password “pass” ‘ partition by list( mod(col_a, 2)) ( partition pt1 values in(0) comment ‘host “DB3”’, partition pt2 values in(1) comment ‘host “DB4”’ ); tbl_a
- 18. Step 2 DB1 tbl_a DB3 tbl_a col_a%2=1col_a%2=0 DB2 DB4 tbl_a Copy table data from DB1 to DB2. (Use mysqldump with “--master-data = 1 or 2” option) tbl_a
- 19. Step 3 Start replication from DB1 to DB2. Wait for resolving replication delay. DB1 tbl_a DB3 tbl_a col_a%2=1col_a%2=0 DB2 DB4 tbl_a tbl_a replication
- 20. Step 4 Stop client access for DB1. Wait for resolving replication delay. Switch client access from DB1 to DB2. DB1 tbl_a DB3 tbl_a col_a%2=1col_a%2=0 DB2 DB4 tbl_a tbl_a replication
- 21. Finish Stop replication on DB2. Remove DB1. DB3 tbl_a col_a%2=1col_a%2=0 DB2 DB4 tbl_a tbl_a
- 22. Pros and Cons of Replication way Pros 1. No need to manage lock size for coping. 2. Support non primary key table. Cons 1. Need to stop writing.
- 23. How to migrate to sharding with Spider using Trigger
- 24. Initial Structure There is 1 MariaDB server without Spider. DB1 tbl_aCreate table tbl_a ( col_a int, col_b int, primary key(col_a) ) engine = InnoDB;
- 25. Step 1 (for migrating) Create table on DB2 and DB3. Then create Spider table on DB1. DB1 tbl_a DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 Create table tbl_a2 ( col_a int, col_b int, primary key(col_a) ) engine = Spider Connection ‘ table “tbl_a”, user “user”, password “pass” ‘ partition by list( mod(col_a, 2)) ( partition pt1 values in(0) comment ‘host “DB2”’, partition pt2 values in(1) comment ‘host “DB3”’ );
- 26. Step 2 Create triggers on DB1. (For copying insert, update and delete. If you use “truncate” for tbl_a, you should better to use other way) DB1 tbl_a DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 delimiter | create trigger tbl_a_i after insert on tbl_a for each row insert into tbl_a2 (a,b) values (new.a, new.b); | create trigger tbl_a_u after update on tbl_a for each row update tbl_a2 set a = new.a, b = new.b where a = old.a; | create trigger tbl_a_d after delete on tbl_a for each row delete from tbl_a2 where a = old.a; | delimiter ;
- 27. Step 3 DB1 tbl_a DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 Insert select from tbl_a to tbl_a2. (Please take care of locking time for tbl_a and tbl_a2.)
- 28. Step 4 DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a Rename table from tbl_a2 to tbl_a. Rename table tbl_a to tbl_a3, tbl_a2 to tbl_a; DB1 tbl_a3 tbl_a
- 30. Pros and Cons of Trigger way Pros 1. No need to stop services. 2. Easy to copy.(Simple command) Cons 1. Impossible to support truncate. 2. Need to manage lock size at coping. 3. Impossible to support non primary key.
- 31. How to migrate to sharding with Spider using Spider function
- 32. Initial Structure There is 1 MariaDB server without Spider. DB1 tbl_aCreate table tbl_a ( col_a int, col_b int, primary key(col_a) ) engine = InnoDB;
- 33. Step 1 (for migrating) Create table on DB2 and DB3. Then create Spider table on DB1. DB1 tbl_a DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 Create table tbl_a2 ( col_a int, col_b int, primary key(col_a) ) engine = Spider Connection ‘ table “tbl_a”, user “user”, password “pass” ‘ partition by list( mod(col_a, 2)) ( partition pt1 values in(0) comment ‘host “DB2”’, partition pt2 values in(1) comment ‘host “DB3”’ );
- 34. Step 2 Create tables on DB1. DB1 tbl_a DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2Create table tbl_a4 ( col_a int, col_b int, primary key(col_a) ) engine = Spider Connection ‘ host “localhost” table “tbl_a3 tbl_a2”, lst “0 2”, user “user”, password “pass” ‘; tbl_a4 tbl_a3 Create table tbl_a3 ( col_a int, col_b int, primary key(col_a) ) engine = InnoDB;
- 35. Step 3 Rename table on DB1. DB1 tbl_a3 DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 Rename table tbl_a3 to tbl_a5, tbl_a to tbl_a3, tbl_a4 to tbl_a; tbl_a tbl_a5
- 36. Step 4 Copy data on DB1. DB1 tbl_a3 DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_a2 Select spider_copy_table(‘tbl_a’, ‘’, ‘’); tbl_a tbl_a5
- 37. Step 5 Rename table on DB1. DB1 tbl_a3 DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_a tbl_aRename table tbl_a2 to tbl_a6, tbl_a5 to tbl_a2, tbl_a to tbl_a7, tbl_a6 to tbl_a; tbl_a7 tbl_a2
- 38. Finish DB1 DB2 tbl_a col_a%2=1col_a%2=0 DB3 tbl_atbl_a Drop table tbl_a2, tbl_a3 and tbl_a7.
- 39. Pros and Cons of Spider function way Pros 1. No need to stop services. 2. Easy to copy.(Simple command. Lock size is managed by Spider) Cons 1. Impossible to support non primary key.
- 40. How to migrate to sharding with Spider using Vertical Partitioning Storage Engine
- 41. Initial Structure There is 1 MariaDB server without Spider. DB1 tbl_a Create table tbl_a ( col_a int, col_b int, primary key(col_a), key idx2(col_b) ) engine = InnoDB;
- 42. Step 1 (for migrating) Create table on DB2 and DB3. Then create tables on DB1. DB1 tbl_a DB2 col_a%2=1col_a%2=0 DB3 Create table tbl_pk ( col_a int, primary key(col_a) ) engine = Spider Connection ‘ table “tbl_pk”, user “user”, password “pass” ‘ partition by list( mod(col_a, 2)) ( partition pt1 values in(0) comment ‘host “DB2”’, partition pt2 values in(1) comment ‘host “DB3”’ ); tbl_pk tbl_pk tbl_pk
- 43. Step 2 Create table on DB4 and DB5. Then create tables on DB1. DB1 tbl_a DB2 col_a%2=1col_a%2=0 DB3 Create table tbl_a3 ( col_a int, col_b int, key idx1(col_a), key idx2(col_b) ) engine = Spider Connection ‘ table “tbl_a2”, user “user”, password “pass” ‘ partition by list( mod(col_b, 2)) ( partition pt1 values in(0) comment ‘host “DB4”’, partition pt2 values in(1) comment ‘host “DB5”’ ); tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2
- 44. Step 3 Create tables on DB1. DB1 tbl_a DB2 col_a%2=1col_a%2=0 DB3 Create table tbl_a3 ( col_a int, col_b int, primary key(col_a), key idx2(col_b) ) engine = VP Comment ‘ ctm “1”, ist “1”, pcm “1”, tnl “tbl_a4 tbl_pk tbl_a2” ‘; tbl_a3 tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2 Create table tbl_a4 ( col_a int, col_b int, primary key(col_a) ) engine = InnoDB; tbl_a4
- 45. Step 4 Rename tables on DB1. DB1 DB2 col_a%2=1col_a%2=0 DB3 tbl_a tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2 Rename table tbl_a4 to tbl_a5, tbl_a to tbl_a4, tbl_a3 to tbl_a; tbl_a5 tbl_a4
- 46. Step 5 Copy data from tbl_a4 to tbl_pk and tbl_a2 on DB1. DB1 DB2 col_a%2=1col_a%2=0 DB3 tbl_a tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2 Select vp_copy_tables(‘table_a’, ‘tbl_a4’, ‘tbl_pk tbl_a2’); tbl_a5 tbl_a4
- 47. Step 6 Alter table tbl_a on DB1. DB1 DB2 col_a%2=1col_a%2=0 DB3 tbl_a tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2Alter table tbl_a comment ‘ pcm “1”, tnl “tbl_pk tbl_a2” ‘; tbl_a5 tbl_a4
- 48. Finish Drop table tbl_a on DB1. DB1 DB2 col_a%2=1col_a%2=0 DB3 tbl_a tbl_pk tbl_a2 tbl_pk tbl_pk DB4 col_b%2=1col_b%2=0 DB5 tbl_a2 tbl_a2 Drop table tbl_a4, tbl_a5;
- 49. Pros and Cons of VP way Pros 1. No need to stop services. 2. Support spiltting by non unique columns. 3. Easy to copy.(Simple command. Lock size is managed by VP) Cons 1. VP storage engine is required. 2. Impossible to support non primary key.
- 50. Thank you for taking your time!!