Amazon RDS for PostgreSQL: What's New and Lessons Learned - NY 2017

Editor's Notes

• #10: Line data type Reg* data types Open prepared transactions
• #20: Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
• #21: Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
• #22: Add a Key for the encrypted snapshot and then show that it needs to be shared for this to work. Note that this doesn’t work with default keys.
• #27: Move data to the same or different database engine ~ Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift Keep your apps running during the migration ~ DMS minimizes impact to users by capturing and applying data changes Start your first migration in 10 minutes or less ~ The AWS Database Migration Service takes care of infrastructure provisioning and allows you to setup your first database migration task in less than 10 minutes Replicate within, to or from AWS EC2 or RDS ~ After migrating your database, use the AWS Database Migration Service to replicate data into your Redshift data warehouses, cross-region to other RDS instances, or back to on-premises
• #28: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #29: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #30: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #31: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #32: Using the AWS Database Migration Service to migrate data to AWS is simple. (CLICK) Start by spinning up a DMS instance in your AWS environment (CLICK) Next, from within DMS, connect to both your source and target databases (CLICK) Choose what data you want to migrate. DMS lets you migrate tables, schemas, or whole databases Then sit back and let DMS do the rest. (CLICK) It creates the tables, loads the data, and best of all, keeps them synchronized for as long as you need That replication capability, which keeps the source and target data in sync, allows customers to switch applications (CLICK) over to point to the AWS database at their leisure.DMS eliminates the need for high-stakes extended outages to migrate production data into the cloud. DMS provides a graceful switchover capability.
• #37: Who would like to see more decoders supported
• #68: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #69: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #70: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #71: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #72: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #73: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #74: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #75: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #76: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #77: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #78: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #79: Quorum system for read/write; latency tolerant Quorum membership changes do not stall writes
• #80: Data is replicated 6 times across 3 Availability Zones Continuous backup to Amazon S3 Continuous monitoring of nodes and disks for repair 10GB segments as unit of repair or hotspot rebalance Storage volume automatically grows up to 64 TB
• #81: Data is replicated 6 times across 3 Availability Zones Continuous backup to Amazon S3 Continuous monitoring of nodes and disks for repair 10GB segments as unit of repair or hotspot rebalance Storage volume automatically grows up to 64 TB
• #82: Data is replicated 6 times across 3 Availability Zones Continuous backup to Amazon S3 Continuous monitoring of nodes and disks for repair 10GB segments as unit of repair or hotspot rebalance Storage volume automatically grows up to 64 TB
• #92: (30 seconds): Let’s take a look at data load performance. With the pgbench benchmark, you first have to load the database. We compared the time it takes to load, vacuum, and build indexes for a scale 10000, or 150GB, pgbench database. As you can see, Amazon Aurora can finish the pgbench initialization phase about 3 times faster than PostgreSQL. Most of the performance difference in load times is due to the database-specific storage optimizations that are key to Amazon Aurora storage – we will dive deeper into those optimizations in a few minutes.
• #93: (1 minute): In all the tests we have shown you, we have tried to tune PostgreSQL to deliver the best possible performance results. One key part of that tuning is to reduce the number of checkpoints by increasing the duration between checkpoints. One consequence of that is it increases recovery time if there is a database failure. This is because when recovering from a crash, PostgreSQL has to start from the last checkpoint – the last time it wrote all dirty pages from memory to storage – and roll forward through all the Write-Ahead Log – or WAL – records written since the last checkpoint. The more WAL to roll forward, the longer recovery will take. With Amazon Aurora, there are no checkpoints needed, so recovery time is independent of checkpoints, and independent of how many transactions are being processed by the database. As you can see in the graph, as we increased the checkpoint time for PostgreSQL, the overall throughput increased, but so did the recovery time. At the best throughput level for PostgreSQL, the recovery time for Aurora was 85X faster than for PostgreSQL, and Aurora delivered more than 92 thousand writes per second compared with just under 70 thousand writes per second from PostgreSQL.
• #94: (1 minute) Let’s first look at some pgbench results. Pgbench is the standard benchmark that is part of the PostgreSQL distribution, and it has several built-in modes. One of those modes is tpcb-like, in which pgbench runs transactions that are very similar to the standard TPC-B transaction. We ran pgbench in tpcb-like mode while increasing the number of concurrent client connections from 256 up to 1,536. We used a 30GB scale 2000 size database, and we ran each test for 60 minutes. As you can see in the graph, PostgreSQL reaches a peak of just under 18 thousand transactions per second at 512 connections, whereas Amazon Aurora continues to scale up as more connections are added, reaching a peak of just over 38 thousand transactions per second at 1,024 connections. The peak-to-peak comparison shows Amazon Aurora delivers more than 2x the throughput of PostgreSQL, and the direct comparison of Amazon Aurora’s peak with the corresponding PostgreSQL result with 1,024 connections shows a ratio of greater than 2 ½ times.
• #95: (30 seconds) In this test, we used sysbench, a benchmark utility often used to compare different database engines. We ran the sysbench write-only benchmark, again while increasing the number of client connections, with a 30GB database. PostgreSQL scales up until reaching more than 47 thousand writes per second at 1,024 connections, then the throughput drops as more connections are added; Amazon Aurora scales up to more than 92 thousand writes per second at 1,536 connections, about 2x more throughput when comparing peak to peak. Compared directly with the PostgreSQL throughput with 1,536 connections, the ratio is more than 2 ½ X.
• #96: (1 minute) It’s important to measure throughput, but it’s also important to measure response time at scale. So, we looked at sysbench response times, with 1,024 concurrent connections. On the graph you can see very different behavior for Amazon Aurora as compared with PostgreSQL: the response times for Aurora are much steadier, with much less variation. More precisely, based on measuring the standard deviations of the two data sets, Amazon Aurora is more than 10x more consistent than PostgreSQL. Also, the average response time is about 2.9x lower. So, Aurora delivers much faster response times with much less variability. You might wonder what’s going on with the PostgreSQL results. What you see is the impact of database checkpoints, which PostgreSQL does to ensure that dirty pages in memory are periodically written to storage to ensure recovery time from a crash isn’t extended too long. During a checkpoint, PostgreSQL will do a lot of writes, which will slow down user transactions, hence the variability in the PostgreSQL response times. ------------ NOTES ONLY – DO NOT USE --------------- On the sysbench response time graph:  Stddev(POPS) is 96.97ms.  Stddev(Manny) is 7.38ms.  13x more consistent (although I prefer “greater than 10x”) Avg(POPS) is 201ms, avg(Manny) is 69ms. So this is really 2.9x lower response times (3x would be 207, but who’s counting…) On the sysbench response time graph:  Stddev(POPS) is 96.97ms.  Stddev(Manny) is 7.38ms.  13x more consistent (although I prefer “greater than 10x”) Avg(POPS) is 201ms, avg(Manny) is 69ms. So this is really 2.9x lower response times (3x would be 207, but who’s counting…)
• #97: (30 seconds): Let’s go back to pgbench to look at consistent perfornance based on throughput. In this graph, higher is better, as we’re showing the throughput while running pgbench in tpcb-like mode. We ran each database at the optimal number of clients to deliver max throughput for that database, and plotted the variability in throughput over time. As you can see, Amazon Aurora is much more consistent, and delivers significantly higher throughput: based on standard deviation, Aurora is about 3x more consistent than PostgreSQL. [On the pgbench throughput graph:  stddev(POPS) is 5080 tps.  Stddev(Manny) is 1395 tps. 3.6x more consistent (again, I hate over precision, so “3x” is my preference).]
• #98: (1 minute): In this test, we compared how each database scales in terms of throughput as the database size scales, using the sysbench write-only workload. As you can see, with a 10GB database, Aurora delivers about 1.5X better throughput; with a 100GB database Aurora delivers about 3X better throughput. Aurora can handle larger databases and workloads significantly better than PostgreSQL.
• #99: Ask about preview environment and other decoders