![@cockroachdb
INSERT INTO test VALUES (1, “ball”, NULL);
INSERT INTO test VALUES (2, NULL, NULL);
CockroachDB: NULL Column Values
Key: /<table>/<index>/<key>[/<column>] Value
/test/primary/1 Ø
/test/primary/1/name “ball”
/test/primary/2 Ø](https://image.slidesharecdn.com/ntjv2nm2shu32mi0an6y-signature-c843040d8820a3033a05f0871e1c74658ce11e5cc7a8a13b162d7fb087be3db8-poli-151105184053-lva1-app6892/85/PostgreSQL-and-CockroachDB-SQL-39-320.jpg)
![@cockroachdb
CREATE UNIQUE INDEX bar ON test (name);
INSERT INTO test VALUES (3, NULL, NULL);
CockroachDB: Unique Indexes (NULL Values)
Key: /<table>/<index>/<key>[/<pkey>] Value
/test/bar/NULL/3 Ø](https://image.slidesharecdn.com/ntjv2nm2shu32mi0an6y-signature-c843040d8820a3033a05f0871e1c74658ce11e5cc7a8a13b162d7fb087be3db8-poli-151105184053-lva1-app6892/85/PostgreSQL-and-CockroachDB-SQL-47-320.jpg)
![@cockroachdb
CREATE UNIQUE INDEX bar ON test (name);
INSERT INTO test VALUES (3, NULL, NULL);
INSERT INTO test VALUES (4, NULL, NULL);
CockroachDB: Unique Indexes (NULL Values)
Key: /<table>/<index>/<key>[/<pkey>] Value
/test/bar/NULL/3 Ø
/test/bar/NULL/4 Ø](https://image.slidesharecdn.com/ntjv2nm2shu32mi0an6y-signature-c843040d8820a3033a05f0871e1c74658ce11e5cc7a8a13b162d7fb087be3db8-poli-151105184053-lva1-app6892/85/PostgreSQL-and-CockroachDB-SQL-48-320.jpg)
PostgreSQL and CockroachDB SQL
- 1. @cockroachdb PostgreSQL meetup, November 2015 CockroachDB presented by Peter Mattis / Co-Founder
- 2. @cockroachdb 1.Overview of CockroachDB 2.SQL Data Model 3.Logical Data Storage 4.Online/Concurrent Schema Change Agenda
- 3. @cockroachdb What is CockroachDB? ■Scale out SQL ■Distributed ■Survivable ■Consistent ■Open source
- 4. @cockroachdb CockroachDB: Architecture ■Layered abstractions ■SQL is starting point ■Distributes at map ■Replicates at physical layer SQL Transactional KV Monolithic Map Raft
- 5. @cockroachdb CockroachDB: Architecture ■Layered abstractions ■SQL is starting point ■Distributes at map ■Replicates at physical layer Transactional KV Monolithic Map Raft GraphSQL
- 6. @cockroachdb CockroachDB: Architecture ■Layered abstractions ■SQL is starting point ■Distributes at map ■Replicates at physical layer Transactional KV Monolithic Map Raft SQL Graph Doc
- 7. @cockroachdb CockroachDB: Architecture ■Layered abstractions ■SQL is starting point ■Distributes at map ■Replicates at physical layer Transactional KV Monolithic Map Raft SQL
- 8. @cockroachdb CockroachDB: Architecture ■Layered abstractions ■SQL is starting point ■Distributes at map ■Replicates at physical layer Transactional KV Monolithic Map Raft SQL Physical
- 13. @cockroachdb ■Tables ■Rows ■Columns SQL Data Model Inventory ID Name Price 1 Glove 1.11 2 Ball 2.22 3 Shirt 3.33 4 Shorts 4.44 5 Bat 5.55 6 Shoes 6.66
- 14. @cockroachdb ■Tables ■Rows ■Columns ■Indexes SQL Data Model Inventory ID Name Price 1 Glove 1.11 2 Ball 2.22 3 Shirt 3.33 4 Shorts 4.44 5 Bat 5.55 6 Shoes 6.66 Name Ball Bat Glove Shirt Shoes Shorts Name_Idx
- 15. @cockroachdb PostgreSQL: Logical Data Storage
- 16. @cockroachdb ■Rows are stored in an unordered heap ■Indexes are btrees ■Primary key is a unique index PostgreSQL: Data Storage
- 17. @cockroachdb CREATE TABLE test ( id INTEGER PRIMARY KEY, name VARCHAR, price FLOAT, ); PostgreSQL: Example Table
- 18. @cockroachdb INSERT INTO test VALUES (1, “ball”, 3.33); PostgreSQL: Logical Data Storage
- 19. @cockroachdb INSERT INTO test VALUES (1, “ball”, 3.33); PostgreSQL: Logical Data Storage Tuple ID (Page# / Item#) Row (0, 1) (1, “ball”, 3.33) test (heap)
- 20. @cockroachdb INSERT INTO test VALUES (1, “ball”, 3.33); PostgreSQL: Logical Data Storage Tuple ID (Page# / Item#) Row (0, 1) (1, “ball”, 3.33) Index Key Tuple ID 1 (0, 1) test (heap)test_pkey (btree)
- 21. @cockroachdb INSERT INTO test VALUES (1, “ball”, 3.33); INSERT INTO test VALUES (2, “glove”, 4.44); PostgreSQL: Logical Data Storage Tuple ID (Page# / Item#) Row (0, 1) (1, “ball”, 3.33) (0, 2) (2, “glove”, 4.44) Index Key Tuple ID 1 (0, 1) 2 (0, 2) test (heap)test_pkey (btree)
- 22. @cockroachdb CockroachDB: Logical Data Storage
- 23. @cockroachdb ■Keys and values are strings ■Monolithic, sorted map CockroachDB: KV
- 24. @cockroachdb Get(key) Put(key, value) ConditionalPut(key, value, expValue) Scan(startKey, endKey) CockroachDB: KV Primitives
- 25. @cockroachdb Get(key) Put(key, value) ConditionalPut(key, value, expValue) Scan(startKey, endKey) Del(key) CockroachDB: KV Primitives
- 26. @cockroachdb ■All tables have a primary key ■One key/value pair per column CockroachDB: Row Storage
- 27. @cockroachdb ■All tables have a primary key ■One key/value pair per column ■Key anatomy: /<table>/<index>/<pkey>/<column> CockroachDB: Row Storage
- 28. @cockroachdb CREATE TABLE test ( id INTEGER PRIMARY KEY, name VARCHAR, price FLOAT, ); CockroachDB: Example Table
- 29. @cockroachdb INSERT INTO test VALUES (1, “ball”, 2.22); CockroachDB: Key Anatomy Key: /<table>/<index>/<key>/<column> Value /test/primary/1/name “ball” /test/primary/1/price 2.22
- 30. @cockroachdb INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); CockroachDB: Key Anatomy Key: /<table>/<index>/<key>/<column> Value /test/primary/1/name “ball” /test/primary/1/price 2.22 /test/primary/2/name “glove” /test/primary/2/price 3.33
- 31. @cockroachdb INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); CockroachDB: Key Anatomy Key: /<table>/<index>/<key>/<column> Value /test/primary/1/name “ball” .../price 2.22 .../2/name “glove” .../price 3.33
- 32. @cockroachdb INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); CockroachDB: Key Anatomy Key: /<table>/<index>/<key>/<column> Value /1000/1/1/1 “ball” .../2 2.22 .../2/1 “glove” .../2 3.33
- 33. @cockroachdb ■Key encoding ■NULL column values ■Unique indexes ■Non-unique indexes CockroachDB: The Details
- 34. @cockroachdb ■Keys and values are strings ■Columns are typed data ■??? CockroachDB: Key Encoding
- 35. @cockroachdb ■NULL indicates value does not exist ■NULL is weird: NULL != NULL CockroachDB: NULL Column Values
- 36. @cockroachdb ■NULL indicates value does not exist ■NULL is weird: NULL != NULL ■CockroachDB: NULL values are not explicitly stored CockroachDB: NULL Column Values
- 37. @cockroachdb INSERT INTO test VALUES (1, “ball”, NULL); CockroachDB: NULL Column Values Key: /<table>/<index>/<key>/<column> Value /test/primary/1/name “ball”
- 38. @cockroachdb INSERT INTO test VALUES (1, “ball”, NULL); INSERT INTO test VALUES (2, NULL, NULL); CockroachDB: NULL Column Values Key: /<table>/<index>/<key>/<column> Value /test/primary/1/name “ball” ??? ???
- 39. @cockroachdb INSERT INTO test VALUES (1, “ball”, NULL); INSERT INTO test VALUES (2, NULL, NULL); CockroachDB: NULL Column Values Key: /<table>/<index>/<key>[/<column>] Value /test/primary/1 Ø /test/primary/1/name “ball” /test/primary/2 Ø
- 40. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); ■Multiple table rows with equal indexed values are not allowed CockroachDB: Unique Indexes
- 41. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); CockroachDB: Unique Indexes Key: /<table>/<index>/<key> Value /test/bar/”ball” 1
- 42. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); CockroachDB: Unique Indexes Key: /<table>/<index>/<key> Value /test/bar/”ball” 1 /test/bar/”glove” 2
- 43. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); INSERT INTO test VALUES (3, “glove”, 4.44); CockroachDB: Unique Indexes Key: /<table>/<index>/<key> Value /test/bar/”ball” 1 /test/bar/”glove” 2 /test/bar/”glove” 3
- 44. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); ■NULL is weird: NULL != NULL CockroachDB: Unique Indexes (NULL Values)
- 45. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (3, NULL, NULL); CockroachDB: Unique Indexes (NULL Values) Key: /<table>/<index>/<key> Value /test/bar/NULL 3
- 46. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (3, NULL, NULL); INSERT INTO test VALUES (4, NULL, NULL); CockroachDB: Unique Indexes (NULL Values) Key: /<table>/<index>/<key> Value /test/bar/NULL 3 /test/bar/NULL 4
- 47. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (3, NULL, NULL); CockroachDB: Unique Indexes (NULL Values) Key: /<table>/<index>/<key>[/<pkey>] Value /test/bar/NULL/3 Ø
- 48. @cockroachdb CREATE UNIQUE INDEX bar ON test (name); INSERT INTO test VALUES (3, NULL, NULL); INSERT INTO test VALUES (4, NULL, NULL); CockroachDB: Unique Indexes (NULL Values) Key: /<table>/<index>/<key>[/<pkey>] Value /test/bar/NULL/3 Ø /test/bar/NULL/4 Ø
- 49. @cockroachdb CREATE INDEX foo ON test (name); ■Multiple table rows with equal indexed values are allowed CockroachDB: Non-Unique Indexes
- 50. @cockroachdb CREATE INDEX foo ON test (name); ■Multiple table rows with equal indexed values are allowed ■Primary key is a unique index CockroachDB: Non-Unique Indexes
- 51. @cockroachdb CREATE INDEX foo ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); CockroachDB: Non-Unique Indexes Key: /<table>/<index>/<key>/<pkey> Value /test/foo/”ball”/1 Ø
- 52. @cockroachdb CREATE INDEX foo ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); CockroachDB: Non-Unique Indexes Key: /<table>/<index>/<key>/<pkey> Value /test/foo/”ball”/1 Ø /test/foo/”glove”/2 Ø
- 53. @cockroachdb CREATE INDEX foo ON test (name); INSERT INTO test VALUES (1, “ball”, 2.22); INSERT INTO test VALUES (2, “glove”, 3.33); INSERT INTO test VALUES (3, “glove”, 4.44); CockroachDB: Non-Unique Indexes Key: /<table>/<index>/<key>/<pkey> Value /test/foo/”ball”/1 Ø /test/foo/”glove”/2 Ø /test/foo/”glove”/3 Ø
- 54. @cockroachdb ■Keys and values are strings ■NULL column values ■Unique indexes ■Non-unique indexes CockroachDB: Logical Data Storage
- 55. @cockroachdb Logical Data Storage PostgreSQL CockroachDB Keys are composite structures Keys are strings Heap storage for rows Required primary key Per-table heap/indexes Monolithic map
- 57. @cockroachdb Schema Change Operations CREATE INDEX foo ON test (col1, col2, …); ALTER TABLE test DROP col1; ALTER TABLE test ADD col3 INTEGER; ...
- 58. @cockroachdb Schema Change (the easy way) 1. Lock table 2. Adjust table data (add column, populate index, etc.) 3. Unlock table
- 59. @cockroachdb Schema Change (the easy way) 1. Apologize for down time 2. Lock table 3. Adjust table data (add column, populate index, etc.) 4. Unlock table
- 60. @cockroachdb Schema Change (the MySQL way) 1. Create new table with altered schema 2. Capture changes from source to the new table 3. Copy rows from the source to the new table 4. Synchronize source and new table 5. Swap/rename source and new table
- 61. @cockroachdb Schema Change (the PostgreSQL way) 1. CREATE INDEX CONCURRENTLY
- 62. @cockroachdb CockroachDB: Schema Change ■TableDescriptor contains table schema ■TableDescriptor replicated on every node ■Distributed atomic updates are difficult ■Distributed locking is difficult ■The easy way isn’t feasible
- 63. @cockroachdb CockroachDB: CREATE INDEX CREATE INDEX foo ON TEST 1. Backfill index entries 2. Add index to TableDescriptor
- 64. @cockroachdb CockroachDB: CREATE INDEX T1 T2 CREATE INDEX foo ON test… INSERT INTO test…
- 65. @cockroachdb CockroachDB: CREATE INDEX CREATE INDEX foo ON TEST 1. Add index to TableDescriptor as write-only 2. Backfill index entries 3. Mark index as read-write
- 66. @cockroachdb CockroachDB: CREATE INDEX T1 T2 CREATE INDEX foo ON test… INSERT INTO test… or UPDATE test… or DELETE FROM test…
- 67. @cockroachdb CockroachDB: CREATE INDEX CREATE INDEX foo ON TEST 1. Add index to TableDescriptor as delete-only 2. Wait for descriptor propagation 3. Mark index as write-only 4. Wait for descriptor propagation 5. Backfill index entries 6. Mark index as read-write
- 68. @cockroachdb Online Schema Change Online schema change is difficult The database should do the heavy lifting
- 69. @cockroachdb The End SQL databases are KV stores on steroids
Editor's Notes
- #4: This is a PostgreSQL meetup, why should I care about CockroachDB? The CockroachDB SQL grammar is based on the Postgres grammar. Postgres is a SQL database. CockroachDB is a distributed SQL database.
- #5: Layered abstractions make it possible to deal with complexity Higher levels can treat lower levels as functional black boxes
- #6: Top two layers are logical Don’t stop at SQL!
- #8: Monolithic sorted map is distributed layer
- #9: RocksDB at physical layer
- #16: This is a brief overview of logical data storage in PostgreSQL. I’m using the term “logical” to refer to how SQL data is mapped down into PostgreSQL structures and distinguishing it from “physical” data storage which is exactly how those structures are implemented.
- #17: The heap structure is unindexed storage of row tuples. Think of it as a hash table where rows are given a unique id at insertion time, except that it is unfortunately not that simple. Tuples (rows) are located by a tuple ID which is composed of a page# and item# within the page. A Btree stores values sorted by key. Btree index key is a tuple of the columns in the index. Value is the row’s tuple ID.
- #18: An example will make this clearer.
- #22: Tuple IDs just happen to be ordered in this example. They are not in general. And tuple IDs are an internal detail of tables and not stable for external usage.
- #38: Row sentinel!
- #39: Row sentinel!
- #40: Row sentinel!
- #41: Value contains any primary key column that is not stored in index key.
- #42: Value contains any primary key column that is not stored in index key.
- #43: Value contains any primary key column that is not stored in index key.
- #44: Value contains any primary key column that is not stored in index key.
- #45: Value contains any primary key column that is not stored in index key.
- #57: Poll audience about experience with production schema change.