In-core compression: how to shrink your database size in several times

www.postgrespro.ru
In-core compression:
how to shrink your database
size in several times
Aleksander Alekseev
Anastasia Lubennikova

Agenda
●
What does Postgres store?
• A couple of words about storage internals
●
Check list for your schema
• A set of tricks to optimize database size
●
In-core block level compression
• Out-of-box feature of Postgres Pro EE
●
ZSON
• Extension for transparent JSONB compression

What this talk doesn’t cover
●
MVCC bloat
• Tune autovacuum properly
• Drop unused indexes
• Use pg_repack
• Try pg_squeeze
●
Catalog bloat
• Create less temporary tables
●
WAL-log size
• Enable wal_compression
●
FS level compression
• ZFS, btrfs, etc

Empty tables are not that empty
●
Imagine we have no data
create table tbl();
insert into tbl select from generate_series(0,1e07);
select pg_size_pretty(pg_relation_size('tbl'));
pg_size_pretty
---------------
???

Empty tables are not that empty
●
Imagine we have no data
create table tbl();
insert into tbl select from generate_series(0,1e07);
select pg_size_pretty(pg_relation_size('tbl'));
pg_size_pretty
---------------
268 MB

Order matters
●
Attributes must be aligned inside the row
Safe up to 20% of space.
create table bad (i1 int, b1 bigint, i1 int);
create table good (i1 int, i1 int, b1 bigint);

NULLs for free*
●
Tuple header size: 23 bytes
●
With alignment: 24 bytes
●
Null mask is placed right after a header
●
Result: up to 8 nullable columns cost nothing
●
Also: buy one NULL, get 7 NULLs for free! (plus
alignment)
* not actually free

Alignment and B-tree
All index entries are 8 bytes aligned
create table good (i1 int, i1 int, b1 bigint);
create index idx on good (i1);
create index idx_multi on good (i1, i1);
create index idx_big on good (b1);

Alignment and B-tree
●
It cannot be smaller, but it can keep more data
●
Covering indexes* may come in handy here
• CREATE INDEX tbl_pkey (i1) INCLUDE (i2)
●
+ It enables index-only scan for READ queries
●
– It disables HOT updates for WRITE queries
*Already in PostgresPro, hopefully will be in PostgreSQL 10

Use proper data types
CREATE TABLE b AS
SELECT 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'::bytea;
select lp_len, t_data from heap_page_items(get_raw_page('b',0));
lp_len | t_data
-------+---------------------------------------------------------
61 |
x4b61306565626339392d396330622d346566382d626236642d3662623962643
33830613131
CREATE TABLE u AS
SELECT 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'::uuid;
select lp_len, t_data from heap_page_items(get_raw_page('u',0));
lp_len | t_data
-------+------------------------------------
40 | xa0eebc999c0b4ef8bb6d6bb9bd380a11

Timetz vs timestamptz
●
timetz: int64 (timestamp) + int32 (timezone)
●
timestamptz: always an int64 in UTC
●
Result: time takes more space then date + time

TOAST
●
Splitting of oversized attributes with an optional
compression
• PGLZ: more or less same (speed, ratio) as ZLIB
• Heuristic: if beginning of the attribute is compressed
well then compress it
• Works out of the box for large string-like attributes

Know your data and your database
●
Use proper data types
●
Reorder columns to avoid padding
●
Pack data into bigger chunks to trigger TOAST

CFS
●
CFS — «compressed file system»
• Out of box (PostgresPro Enterprise Edition)
decompress
compress

Layout changes
●
Postgres layout ●
CFS layout

CFS usage
CREATE TABLESPACE cfs LOCATION
'/home/tblspc/cfs' with (compression=true);
SET default_tablespace=cfs;
CREATE TABLE tbl (x int);
INSERT INTO tbl VALUES (generate_series(1, 1000000));
UPDATE tbl set x=x+1;
SELECT cfs_start_gc(4); /* 4 — number of workers */

Pgbench performance
●
pgbench -s 1000 -i
• 2 times slower
• 98 sec → 214 sec
●
database size
• 18 times smaller
• 15334 MB → 827 MB
●
pgbench -c 10 -j 10 -t 10000
• 5% better
• 3904 TPS → 4126 TPS

Comparison of
compression algoritms
Configuration Size (Gb) Time (sec)
no compression 15.31 92
snappy 5.18 99
lz4 4.12 91
postgres internal lz 3.89 214
lzfse 2.80 1099
zlib (best speed) 2.43 191
zlib (default level) 2.37 284
zstd 1.69 125
pgbench -i -s 1000

CFS pros
●
Good compression rate:
• All information on the page is compressed including
headers
●
Better locality:
• CFS always writes new pages sequentially
●
Minimal changes in Postgres core:
• CFS works at the lowest level
●
Flexibility:
• Easy to use various compression algorithms

CFS cons
●
Shared buffers utilization:
• Buffer cache keeps pages uncompressed
●
Inefficient WAL and replication:
• Replica has to perform compression and GC itself
●
Fragmentation
• CFS needs its own garbage collector

ZSON
●
An extension for transparent JSONB compression
●
A dictionary of common strings is created based
on your data (re-learning is also supported)
●
This dictionary is used to replace strings to 16 bit
codes
●
Data is compressed in memory and on the disk
●
In some cases it gives 10% more TPS
●
●
https://github.com/postgrespro/zson

JSONB Problems
●
Redundancy
●
Disk space
●
Memory
●
=> IO & TPS

The Idea
●
Step 1 — replace common strings to 16 bit codes
●
Step 2 — compress using PGLZ as usual

zson_learn
zson_learn(
tables_and_columns text[][],
max_examples int default 10000,
min_length int default 2,
max_length int default 128,
min_count int default 2
)
Example:
select zson_learn('{{"table1", "col1"}, {"table2", "col2"}}');

Encoding
// VARHDRSZ
// zson_version [uint8]
// dict_version [uint32]
// decoded_size [uint32]
// hint [uint8 x PGLZ_HINT_SIZE]
// {
//skip_bytes [uint8]
//... skip_bytes bytes ...
//string_code [uint16], 0 = no_string
// } *

Thank you for your attention!
Any questions?
●
https://postgrespro.com/
●
a.lubennikova@postgrespro.ru
●
a.alekseev@postgrespro.ru

CFS parameters
●
cfs_gc_workers = 1
• Number of background workers performing CFS
garbage collection
●
cfs_gc_threashold = 50%
• Percent of garbage in the file after which
defragmentation begins
●
cfs_gc_period = 5 seconds
• Interval between CFS garbage collection iterations
●
cfs_gc_delay = 0 milliseconds
• Delay between files defragmentation

In-core compression: how to shrink your database size in several times

More Related Content

What's hot (20)

Similar to In-core compression: how to shrink your database size in several times (20)

More from Aleksander Alekseev (12)

Recently uploaded (20)

In-core compression: how to shrink your database size in several times