PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki

https://www.2ndQuadrant.com
PostgreSQL Database Migration and
Maintenance Avoiding Long Downtime
Koichi Suzuki
PGConf.Asia 2019

PostgreSQL Now
● Features
○ Matured for very wide range of applications
● More production use case
○ Important business scenes
● More chance of database maintenance
○ Version up
○ Database design change
■ Physical
■ Logical
● Business Requirement (24/7 for example)
○ Less acceptance of long maintenance window

Database Migration and Design Change
● Column data type change
○ INT -> BIGINT
○ INT -> NUMERIC
○ VARCHAR -> TEXT
Accomodate more records to a table
Business Growth
ALER TABLE?
COPY?
CREATE TABLE … SELECT?

Partitioning

Migration of too many tables
Tables > 500,000
pg_dump too slow and hard to run with on-going
business process

Most of these operations are blocking
Typically take days
Cannot simply perform in production environment
Many use case require minimum down time
(maintenance window), for example,
< 15min.
General Issue

● Application characteristics
● Data characteristics
Deeply depends upon use case speciﬁcs
Hope following slides is applicable to many other cases.
How to avoid long maintenance window

Partitioning an existing table
Usual process:
1. CREATE TABLE PARTITIONED …
2. COPY from old table to new table
Typical application/data characteristics:
● Log type: can use creation timestamp as partitioning
key
● Data are not updated (may be deleted though)
● Has another primary key which is used as foreign key
from other tables, but not declared explicitly.

Apps can continue to run.
Partitioning Steps
1. Create new table with same column deﬁnition
a. Creation timestamp as the partitioning key
b. Exclude for PKEY/Unique constraint
2. Create partitions as needed
3. Copy row from original table
4. Lock original/new table
5. Do last copy
6. Rename original/new table
7. Unlock original/new table
Maintenance Window,
requres apps to stop.

Original Table

New Table

Partitions

First Data Copy: Don’t need to stop Apps.

Last Step: Need to stop Apps.

Maintain Unique Constraint
Partitioning key must be a part of unique constraints
Maintaining uniqueness is application’s responsibility
1. INSERT
○ Use of sequence/other default value
2. UPDATE
○ Use RULE/TRIGGER to prohibit unique value update
All these depends upon application/date characteristics and/or
requirements.
Unique constraint in each partition may reduce a risk.

Discussed in the unconference, lead by Ibrar Ahmed on Sunday.
Adding Unique Constraint

Maintain Unique Constraint

Change Sequence Datatype
INTEGER to BIGINT: Beyond 4G limit

Change Sequence Datatype (2)

Change Sequence Datatype (3)

Migrate too many tables
● Some database have huge number of tables
> 500,000
● Migration by pg_dump is not practical
○ Takes too long
● Upgrade from very old version
< even 9.0

Why FDW is good
● No speciﬁc feature/extension on the source
● Only libpq is needed
● Need postgres_fdw extension at the target

How FDW works
Source PostgreSQL
libpq libpq
Target PostgreSQL
postgres_fdw

Installing and setup postgres_fdw
May need more, depending
upon target database, for
example, port and passwd.

Table migration (1)
● We can import all the remote schema
● No need to create each local table
Create Foreign Table only:
Actual data is still at remote
server.
Need to tweak
max_locks_per_transaction GUC
setting to import more tables in
single statement.

Table Migration (2)
● Need to issue CREATE TABLE to import the data to
local server
To do this in single transaction for speedup, need
to tweak max_locks_per_transaction GUC setting.

Evaluation
● Comparison with pg_dump
● 300,000 tables
● 1,000 rows for each table
● 3 columns:
○ one: integer,
○ two: varchar(32),
■ random length, random value

Table template

Table data

Environment
SSD (SATA)
PostgreSQL 9.3PostgreSQL 11.4
postgres_fdw
pg_dump
postgres 9.3 databasepostgres 11.4 database

Result
pg_dump dump tables at PostgreSQL 9.3 1:51
pg_dump load tables to PostgreSQL 11.4 7:04
FDW Import table schema 1’18”
FDW Import all the data 2:29
We need to consider the duration needed to copy
pg_dump data from remote to local. In this case, the size
of pg_dump output was about 11Gig.

Remarks/Summary
● No general steps to avoid long down time in
database migration
● Need to observe characteristics of application and
data itself
● Find what is dynamic and what is static
○ Beware DELETE and UPDATE
● Leverage static characteristics of the data
● Beware requirements/constraints
○ Find if you can ease requirements
■ May need to work with application
■ Plan carefully: schedule and resource
○ May need compromise

Possible PostgreSQL/Relevant Features
● Pkey/unique constraint over partitions
○ Ibrar Ahmed is working.
● View as partition
● Concurrent ALTER TABLE
● Logical replication

koichi@2ndquadrant.com
Thank you very much!

Change enum data type
Enum type: customer_class
class1
class2
class2
priority0
priority1
normal0
normal1
internal0
internal1
Business Rule Change
->
Sometimes need both column for a while

PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki

More Related Content

What's hot (20)

Similar to PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki (20)

More from Equnix Business Solutions (20)

Recently uploaded (20)

PGConf.ASIA 2019 Bali - PostgreSQL Database Migration and Maintenance - Koichi Suzuki