NoSQL store everyone ignored - Postgres Conf 2021
0 likes72 views
The document discusses the evolution and benefits of using PostgreSQL's hstore and jsonb data types for semi-structured data storage, particularly in relation to Friendfeed's back-end system. It outlines the development history of hstore, its features, and how it allows for flexible data models, while also contrasting it with jsonb's hierarchical structure and indexing capabilities. The text highlights practical applications, indexing strategies, and performance considerations for managing large-scale data with PostgreSQL.
![JSONB-JI
S
SELECT content#>>’{text}' as txt
FROM tweets
WHERE (content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb;](https://image.slidesharecdn.com/pgconf2021-210810204135/85/NoSQL-store-everyone-ignored-Postgres-Conf-2021-43-320.jpg)
![MAT
CH
ING TAGS
SELECT content#>>’{text}' as txt
FROM tweets
WHERE (content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb;](https://image.slidesharecdn.com/pgconf2021-210810204135/85/NoSQL-store-everyone-ignored-Postgres-Conf-2021-46-320.jpg)
![Complex S
ea
r
c
CREATE INDEX idx_gin_rt_hashtags
ON tweets
USING GIN ((content#>'{retweeted_status,entities,hashtags}') jsonb_ops);
SELECT content#>'{text}' as txt
FROM tweets
WHERE (
(content#>'{entities,hashtags}') @> '[{"text": “python"}]'::jsonb
OR
(content#>'{retweeted_status,entities,hashtags}') @> '[{"text": “postgres"}]'::jsonb
);](https://image.slidesharecdn.com/pgconf2021-210810204135/85/NoSQL-store-everyone-ignored-Postgres-Conf-2021-48-320.jpg)
![JSONB + PA
RT
ITIOn + INDEXING
EXPLAIN SELECT content#>'{text}' as txt
FROM part_tweets
WHERE (content#>'{entities,hashtags}') @> '[{"text": "postgres"}]'::jsonb;
QUERY PLAN
----------------------------------------------------------------------------------------------------------
-
Append (cost=24.26
.
.
695.46 rows=131 width=32)
-
>
Bitmap Heap Scan on part_tweets_0 (cost=24.26
.
.
150.18 rows=34 width=32)
Recheck Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Index Scan on part_tweets_0_expr_idx (cost=0.00
.
.
24.25 rows=34 width=0)
Index Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Heap Scan on part_tweets_1 (cost=80.25
.
.
199.02 rows=32 width=32)
Recheck Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Index Scan on part_tweets_1_expr_idx (cost=0.00
.
.
80.24 rows=32 width=0)
Index Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Heap Scan on part_tweets_2 (cost=28.25
.
.
147.15 rows=32 width=32)
Recheck Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Index Scan on part_tweets_2_expr_idx (cost=0.00
.
.
28.24 rows=32 width=0)
Index Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Heap Scan on part_tweets_3 (cost=76.26
.
.
198.46 rows=33 width=32)
Recheck Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
-
>
Bitmap Index Scan on part_tweets_3_expr_idx (cost=0.00
.
.
76.25 rows=33 width=0)
Index Cond: ((content #> '{entities,hashtags}'
:
:
text[]) @> '[{"text": "postgres"}]'
:
:
jsonb)
(17 rows)](https://image.slidesharecdn.com/pgconf2021-210810204135/85/NoSQL-store-everyone-ignored-Postgres-Conf-2021-53-320.jpg)
![JSONB + PA
RT
ITIOn + INDEXING
EXPLAIN SELECT content#>'{text}' as txt FROM tweets WHERE (
(content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb
OR
(content#>'{retweeted_status,entities,hashtags}') @> '[{"text": "python"}]'::jsonb
);](https://image.slidesharecdn.com/pgconf2021-210810204135/85/NoSQL-store-everyone-ignored-Postgres-Conf-2021-54-320.jpg)
NoSQL store everyone ignored - Postgres Conf 2021
- 1. The NOSQL ST ORE Ev er yone IGNORED By Zohaib Sibte H as san @ DOorD a
- 2. About mE Zohaib Sibte Hassan @zohaibility Dad, engineer, tinkerer, love open source & working for DoorDash!
- 3. At DoorDash we use Postgres! DoorD a
- 4. Hi st ory 2009 - Friend Feed blog
- 5. Hi st ory 2011 - I discovered HSTORE and blogged about it
- 6. Hi st ory 2012 - I revisited imagining FriendFeed on Postgres & HSTORE
- 7. Hi st ory 2015 - Talk with same title in Dublin
- 8. Our Roadmap Today •A brief look at FriendFeed use - case •Warming up with HSTORE •Taking it to next level: •JSONB •Complex yet simple queries •Partitioning our documents
- 9. Po st gr es IS ALWAYs ev olvING •Robust schemaless - types: •Array •HSTORE •XML •JSON & JSONB •Improved storage engine •Improved Foreign Data Wrappers •Partitioning support
- 11. HS TORE May 2003 - First version of store
- 12. HS TORE •May 16, 2003 - f i rst (unpublished) version of hstore for PostgreSQL 7.3 •Dec, 05, 2006 - hstore is a part of PostgreSQL 8.2 •May 23, 2007 - GIN index for hstore, PostgreSQL 8.3 •Sep, 20, 2010 - Andrew Gierth improved hstore, PostgreSQL 9.0 •May 24, 2013 - Nested hstore with array support, key - > value model - > document - based model (Stay tuned for more on this)
- 13. HS TORE •Benef i ts: •Provides a f l exible model for storing a semi - structured data in Postgres. •Binary represented so extremely fast! Selecting f i elds or properties. •GIN and GiST indexes can be used! •Drawbacks: •Too f l at! Doesn't support tree - like structures (e.g. JSON introduced in 2006, 3 years after store)
- 14. Enou gh TH EORY L et 's build som et hing s er ious
- 15. F ri en dFEED U SI NG SQL To BUILD NoSQL • https: / / backchannel.org/blog/friendfeed - schemaless - mysql
- 16. WHY F RI EN DFEED? •Good example of understanding available technology and problem at hand. •Did not cave in to buzzword, and started using something less known/reliable. •Large scale problem with good example on how modern SQL tooling solves the problem. •Using tool that you are comfortable with. •Read blog post!
- 18. F RI EN DFEED { "id": "71f0c4d2291844cca2df6f486e96e37c", "user_id": "f48b0440ca0c4f66991c4d5f6a078eaf", "feed_id": "f48b0440ca0c4f66991c4d5f6a078eaf", "title": "We just launched a new backend system for FriendFeed!", "link": "http: / / friendfeed.com/e/71f0c4d2-2918-44cc - a2df-6f486e96e37c", "published": 1235697046, "updated": 1235697046, }
- 19. F RI EN DFEED CREATE TABLE entities ( added_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, id BINARY(16) NOT NULL, updated TIMESTAMP NOT NULL, body MEDIUMBLOB, UNIQUE KEY (id), KEY (updated) ) ENGINE=InnoDB;
- 20. HOW CA N WE INDEX? { "id": "71f0c4d2291844cca2df6f486e96e37c", "user_id": "f48b0440ca0c4f66991c4d5f6a078eaf", "feed_id": "f48b0440ca0c4f66991c4d5f6a078eaf", "title": "We just launched a new backend system for FriendFeed!", "link": "http://friendfeed.com/e/71f0c4d2-2918-44cc-a2df-6f486e96e37c", "published": 1235697046, "updated": 1235697046, }
- 21. F RI EN DFEED INDEXING CREATE TABLE index_user_id ( user_id BINARY(16) NOT NULL, entity_id BINARY(16) NOT NULL UNIQUE, PRIMARY KEY (user_id, entity_id) ) ENGINE=InnoDB; •Create tables for each indexed f i eld. •Have background workers to populate newly created index. •Complete language framework to ensure documents are indexed as they are inserted.
- 23. HS TORE The K ey -Value Store Ev er yone Ignored
- 24. HS TORE
- 25. HS TORE CREATE TABLE feed ( id varchar(64) NOT NULL PRIMARY KEY, doc hstore );
- 26. HS TORE INSERT INTO feed VALUES ( 'ff923c93-7769-4ef6-b026-50c5a87a79c5', 'id=>zohaibility, post=>hello'::hstore );
- 27. HS TORE SELECT doc - > 'post' as post, doc - > 'undef i ned_f i eld' as undef i ned FROM feed WHERE doc - > 'id' = 'zohaibility'; post | undef i ned -------+----------- hello | (1 row)
- 28. HS TORE EXPLAIN SELECT * FROM feed WHERE doc - > 'id' = 'zohaibility'; QUERY PLAN ------------------------------------------------------- Seq Scan on feed (cost=0.00 . . 1.03 rows=1 width=178) Filter: ((doc - > 'id' : : text) = 'zohaibility' : : text) (2 rows)
- 29. INDEXING HS TORE CREATE INDEX feed_user_id_index ON feed ((doc->'id'));
- 30. HS TORE ❤ GI S CREATE INDEX feed_gist_idx ON feed USING gist (doc);
- 31. HS TORE ❤ GI S SELECT doc->'post' as post, doc->'undefined_field' as should_be_null FROM feed WHERE doc @> ‘id=>zohaibility'; post | undef i ned -------+----------- hello | (1 row)
- 32. R EI MA GI NING Fr ei ndFEED CREATE TABLE entities ( id BIGINT PRIMARY KEY, updated TIMESTAMP NOT NULL, body HSTORE, … ); CREATE TABLE index_user_id ( user_id BINARY(16) NOT NULL, entity_id BINARY(16) NOT NULL UNIQUE, PRIMARY KEY (user_id, entity_id) ) ENGINE=InnoDB; CREATE INDEX CONCURRENTLY entity_id_index ON entities ((body->’entity_id’));
- 35. WHY JSON? •Well understood, and goto standard for almost everything on modern web. •“Self describing”, hierarchical, and parsing and serialization libraries for every programming language •Describes a loose shape of the object, which might be necessary in some cases.
- 36. TW E ET s
- 37. TW E ET S TABLE CREATE TABLE tweets ( id varchar(64) NOT NULL PRIMARY KEY, content jsonb NOT NULL );
- 38. B AS IC QU ER Y SELECT "content"->'text' as txt, "content"->'favorite_count' as cnt FROM tweets WHERE “content"->'id_str' == ‘…’ And Y ES you ca n index th is!!!
- 39. PE EK IN INTO ST RU CT URE SELECT * FROM tweets WHERE (content->>'favorite_count')::integer >= 1;
- 40. 😭 EXPLAIN SELECT * FROM tweets WHERE (content->'favorite_count')::integer >= 1; QUERY PLAN ------------------------------------------------------------------ Seq Scan on tweets (cost=0.00 . . 2453.28 rows=6688 width=718) Filter: (((content - > > 'favorite_count' : : text)) : : integer > = 1) (2 rows)
- 41. B AS IC INDEXING CREATE INDEX fav_count_index ON tweets (((content->’favorite_count')::INTEGER));
- 42. B AS IC INDEXING EXPLAIN SELECT * FROM tweets WHERE (content->'favorite_count')::integer >= 1; QUERY PLAN ----------------------------------------------------------------------------------- Bitmap Heap Scan on tweets (cost=128.12 . . 2297.16 rows=6688 width=718) Recheck Cond: (((content - > 'favorite_count' : : text)) : : integer > = 1) - > Bitmap Index Scan on fav_count_index (cost=0.00 . . 126.45 rows=6688 width=0) Index Cond: (((content - > 'favorite_count' : : text)) : : integer > = 1) (4 rows)
- 43. JSONB-JI S SELECT content#>>’{text}' as txt FROM tweets WHERE (content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb;
- 44. JSON OP ERA TO R
- 46. MAT CH ING TAGS SELECT content#>>’{text}' as txt FROM tweets WHERE (content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb;
- 47. INDEXING CREATE INDEX idx_gin_hashtags ON tweets USING GIN ((content#>'{entities,hashtags}') jsonb_ops);
- 48. Complex S ea r c CREATE INDEX idx_gin_rt_hashtags ON tweets USING GIN ((content#>'{retweeted_status,entities,hashtags}') jsonb_ops); SELECT content#>'{text}' as txt FROM tweets WHERE ( (content#>'{entities,hashtags}') @> '[{"text": “python"}]'::jsonb OR (content#>'{retweeted_status,entities,hashtags}') @> '[{"text": “postgres"}]'::jsonb );
- 49. JSONB + ECO SY ST E TH E POW ER OF AL CH EM Y
- 50. JSONB + TSVE CT OR CREATE INDEX idx_gin_tweet_text ON tweets USING GIN (to_tsvector('english', content->>'text') tsvector_ops); SELECT content->>'text' as txt FROM tweets WHERE to_tsvector('english', content->>'text') @@ to_tsquery('english', 'python');
- 51. JSONB + PA RT ITIOn CREATE TABLE part_tweets ( id varchar(64) NOT NULL, content jsonb NOT NULL ) PARTITION BY hash (md5(content->>’user'->>'id')); CREATE TABLE part_tweets_0 PARTITION OF part_tweets FOR VALUES WITH (MODULUS 4, REMAINDER 0); CREATE TABLE part_tweets_1 PARTITION OF part_tweets FOR VALUES WITH (MODULUS 4, REMAINDER 1); CREATE TABLE part_tweets_2 PARTITION OF part_tweets FOR VALUES WITH (MODULUS 4, REMAINDER 2); CREATE TABLE part_tweets_3 PARTITION OF part_tweets FOR VALUES WITH (MODULUS 4, REMAINDER 3);
- 52. JSONB + PA RT ITIOn + INDEXING CREATE INDEX pidx_gin_hashtags ON part_tweets USING GIN ((content#>'{entities,hashtags}') jsonb_ops); CREATE INDEX pidx_gin_rt_hashtags ON part_tweets USING GIN ((content#>'{retweeted_status,entities,hashtags}') jsonb_ops); CREATE INDEX pidx_gin_tweet_text ON tweets USING GIN (to_tsvector('english', content->>'text') tsvector_ops); INSERT INTO part_tweets SELECT * from tweets;
- 53. JSONB + PA RT ITIOn + INDEXING EXPLAIN SELECT content#>'{text}' as txt FROM part_tweets WHERE (content#>'{entities,hashtags}') @> '[{"text": "postgres"}]'::jsonb; QUERY PLAN ---------------------------------------------------------------------------------------------------------- - Append (cost=24.26 . . 695.46 rows=131 width=32) - > Bitmap Heap Scan on part_tweets_0 (cost=24.26 . . 150.18 rows=34 width=32) Recheck Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Index Scan on part_tweets_0_expr_idx (cost=0.00 . . 24.25 rows=34 width=0) Index Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Heap Scan on part_tweets_1 (cost=80.25 . . 199.02 rows=32 width=32) Recheck Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Index Scan on part_tweets_1_expr_idx (cost=0.00 . . 80.24 rows=32 width=0) Index Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Heap Scan on part_tweets_2 (cost=28.25 . . 147.15 rows=32 width=32) Recheck Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Index Scan on part_tweets_2_expr_idx (cost=0.00 . . 28.24 rows=32 width=0) Index Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Heap Scan on part_tweets_3 (cost=76.26 . . 198.46 rows=33 width=32) Recheck Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) - > Bitmap Index Scan on part_tweets_3_expr_idx (cost=0.00 . . 76.25 rows=33 width=0) Index Cond: ((content #> '{entities,hashtags}' : : text[]) @> '[{"text": "postgres"}]' : : jsonb) (17 rows)
- 54. JSONB + PA RT ITIOn + INDEXING EXPLAIN SELECT content#>'{text}' as txt FROM tweets WHERE ( (content#>'{entities,hashtags}') @> '[{"text": "python"}]'::jsonb OR (content#>'{retweeted_status,entities,hashtags}') @> '[{"text": "python"}]'::jsonb );
- 55. LIMIT IS YOUR IMA GI NATION Don’t Und er E st imate th e pow er of
- 57. TH ANK YOU!