Large Table Partitioning with PostgreSQL and Django

With great data(bases) comes
great responsibility

Large table
partitioning with
PostgreSQL and
Django
Paolo Romolini, Caterina Magini
8 December 2020 - Postgres Build 2020

© Copyright EnterpriseDB Corporation, 2020. All rights reserved.3
• Physicist and major science and technology enthusiast
• In 2ndQuadrant from 2018 to 2020 as DBA and Support Engineer
• Now in EDB
About us
• Web developer
• Python and Django enthusiast
• Customer Portal developer

Summary
• Our use case
• The problem
• Declarative partitioning as possible solution
• The adopted solution

• Facilitate support engineers and
customers job
• Independent management of
companies and users
• Software and repositories
management
• Knowledge base
• Support Ticketing System
The Customer Portal

• Open source
• Dry principle
• MVT logic
• Excellent documentation
• Django REST Framework
Django

• Django.contrib.postgres
• JsonB and Array Fields
• Full Text Search
• PostgreSQL specific model indexes
• The Django Project uses it
• https://github.com/django/djangoproject.com
PostgreSQL and Django

Triggered by ticket events
• Pushover notifications
• Mattermost chats
• Emails
The notifications system

class EmailMessage(models.Model):
subject = models.CharField(max_length=1024)
to_recipients = ArrayField(models.CharField(max_length=256))
message = models.TextField()
from_email = models.CharField(max_length=256)
metadata = models.JSONField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True, null=True)
[...]
The Email Messages app model

• The table was growing too much
• It needed maintenance
Working in DevOps helps!
The Database Table is too large!

• When an application gets deployed to production early on the performance is great
• After a while, the database starts to get bigger and queries slow down...
• ...and at a certain point, you realize the data is the bottleneck
Table partitioning is a good solution
The DBA’s opinion
A typical scenario

Declarative
partitioning as
possible
solution

Declarative partitioning in PostgreSQL
Horizontal partitioning: technique for the vertical scalability of a database
Before PostgreSQL 10 From PostgreSQL 10
• Early “partitioning” introduced in
PostgreSQL 8.1
• Heavily based on relation inheritance
• Constraints to define the partitions,
rules and trigger to route data
• Required a lot of maintenance
• Declarative partitioning
• Simplified syntax
• Partitioned tables = new type of
relationship

The partitioning evolution in PostgreSQL
<=PG 9.6 PG 10 PG 11 PG 12 PG 13
Table inheritance Declarative table
partitioning syntax
Hash partition Partition pruning
enhancements
Partition wise join
enhancements
CHECK constraints Support indexes on
partitions tables
Foreign Keys to
reference
partitioned tables
Partition pruning
improvements
Triggers FOR EACH ROW
Trigger for
partitioned tables
Faster COPY for
partitioned tables
Before ROW-LEVEL
trigger

● Divide et impera
○ Smaller tables, smaller indexes
● Easier maintenance
● Healthier delete operations and bulk loads
○ Remove a table is faster than removing a lot of records
● Overall performance gains
○ Impact of seqscans
Why partitioning?

• RANGE
• LIST
• HASH
Partitioning can also be multi-level
Partitioning strategies
Supported partitioning methods:

Multi-level partitioning
Courtesy of Muhammad Haroon (https://www.2ndquadrant.com/en/blog/scaling-iot-time-series-data-postgres-bdr/)

CREATE TABLE email_messages (
id serial NOT NULL,
subject varchar(1024) NOT NULL,
message text NOT NULL,
from_email varchar(256) NOT NULL,
html_message text NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NULL,
...
send_to_multiple_recipients boolean NULL,
PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);
How does it work?
Partitioned table

CREATE TABLE email_messages_2020_11 PARTITION OF email_messages
FOR VALUES FROM (‘2020-11-01 01:00:00+01’) TO (‘2020-12-01 01:00:00+01’)
CREATE TABLE email_messages_2020_11 PARTITION OF email_messages
FOR VALUES FROM (‘2020-11-01 01:00:00+01’) TO (‘2020-12-01 01:00:00+01’)
How does it work?
Partitions

The list of columns or expressions that define the partitions bound:
• Range
Can have more > 1 column
• Hash
• List Can have only 1 column/expression
The partitioning key
PARTITION BY {RANGE | LIST | HASH}
({ column_name | (expression )} [ COLLATE collation ] [ opclass ] [,...])

Indexes, Primary Keys and Foreign Keys support
● CREATE INDEX on partitioned tables locally-partitioned indexes
● Support for UNIQUE and PRIMARY KEY constraints
○ Partition key has to be part of the PRIMARY KEY
● FOREIGN KEYs
○ Partitioned tables can REFERENCE other tables
○ Partitioned tables can be REFERENCED by other tables

Testing the procedure in QA
Once we understood that partitioning could be the solution, we just had to test it.
• QA environment with PostgreSQL 10
• Large table referenced by another table
• Can’t turn a not partitioned in table into a partitioned one → New table
• Most of the data in the old table are historical, no need to migrate them immediately
• LOCK the old table during the deploy
• Simulate traffic… and check that everything is OK
Needs to upgrade to PG 12

No support for table partitioning in Django
• Third-party django applications?
• Executing SQL query without Django?
Which way to take?

• PostgreSQL declarative partitioning is transparent to Django
• Doing it using Django tools
Django migrations!
The way we have chosen

• A way to track django models changes
• They are applied against the database
• You can revert them
Django migrations
$ python manage.py showmigrations email_messages
email_messages
[X] 0001_initial
[X] 0002_auto_20200716_0837

Specify a different table name
class EmailMessage(models.Model):
[...]
created_at = models.DateTimeField(auto_now_add=True, null=True)
class Meta:
db_table = “email_messages”
Django default database table names:
"%s_%s" % (self.app_label, self.model_name)

Django migrations with custom SQL
class Migration(migrations.Migration):
[...]
operations = [
migrations.RunSQL(
sql=[(PARTITIONED_TABLES_SQL],
reverse_sql=[DROP_PARTITONED_TABLES_SQL],
)
]
Run the sqlmigrate command

The partition key of the Django migration
PARTITIONED_TABLES_SQL = """
CREATE TABLE "email_messages" (
"id" serial NOT NULL PRIMARY KEY,
[...]
) PARTITION BY RANGE
(created_at);
"""
PARTITIONED_TABLES_SQL = """
CREATE TABLE "email_messages" (
"id" serial NOT NULL,
[...]
PRIMARY KEY (id, created_at)
) PARTITION BY RANGE
(created_at);
"""
→

Create the partitions with Django
• Create a new django management command
• Schedule the command as a cron job
• Add another django migration
We are ready to test it in QA!

Deployment in production
• The deployment ran smoothly
• The old table was attached to the new partition
Success!

Conclusion
• The declarative PostgreSQL partitions is good
• Django is a great framework
• DevOps
• That’s one way to do it, not the way!

Follow us:
https://www.linkedin.com/in/paoloromolini/
https://www.linkedin.com/in/caterina-magini/
Tweet:
#PostgresBuild2020
#PostgreSQL
#Django

Large Table Partitioning with PostgreSQL and Django

More Related Content

What's hot (20)

Similar to Large Table Partitioning with PostgreSQL and Django (20)

More from EDB (20)

Recently uploaded (20)

Large Table Partitioning with PostgreSQL and Django