Overcoming write availability challenges of PostgreSQL
Download as PPTX, PDF0 likes535 views
The document discusses the challenges and solutions for achieving high write availability in PostgreSQL, particularly through logical replication methods. It highlights EDB Replicate's capabilities, using Apache Kafka for fault-tolerant and scalable data replication, as well as its benefits for mission-critical applications. The presentation emphasizes the importance of addressing business risks associated with data availability in today's multi-cloud environments.
Overcoming write availability challenges of PostgreSQL
- 1. Overcoming Write Availability Challenges of PostgreSQL Bhavani Rao, John Dalton Hosted by: Molly Halsey
- 2. ● Slides and recording will be available in next 48 hours ● Submit questions via Zoom – will be answering at end Welcome
- 3. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.3 Agenda ● High write availability ● Challenges with logical replication ● Solution ● Features and benefits ● Use cases ● Next steps
- 4. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.4 High Availability Read Availability Write Availability What is write availability? • Redundant DB and infra for tolerating failures • Multiple levels of redundancy for higher avail HA architecture tailored to applications that have a high read to write ratio HA architecture designed for applications with a high write to read ratio
- 5. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.5 Replication to active writable standby Write availability • Fast failover to secondary database • Application writing only to one database • Typically deployed within one region • Often required for financial applications M2M1 App Replication Connects to M2 when M1 fails
- 6. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.6 • Provider: grounded in PostgreSQL, replication, or other DBMS tech • License: commercial or open source • Support: self, community, or vendor • Deployment: private, public, or multi- cloud, single or multi-region Database replication considerations Many factors to consider • Ecosystem: other database and systems in your enterprise infrastructure • Architecture: centralized or distributed • Change capture mechanism: invasive or extension • Replication method: physical or logical, batch or continuous
- 7. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.7 • Fine-grained control over replicating changes to data objects, from selected rows to full database • Node can be on different versions of PostgreSQL • Bi-directional replication, supports writes to multiple database nodes • Interoperability with other database technologies using ANSI SQL standards Replication methods Logical • Binary data in write ahead logs (WAL) from all schemas in active DB sent to replicas byte for byte • Each node on same major PostgreSQL version • Application writes to active database are replicated to read only replicas • Asynchronous and synchronous replication supported Physical Best for Write AvailabilityBest for Read Availability
- 8. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.8 Challenges with logical replication tools • Service must be co-located on same node as PostgreSQL server • Has single point of failure • Dependent on vendor provided or forked PostgreSQL • Limited to DML only changes • Difficult to scale horizontally
- 9. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.9 • Vertical scalability is always limited • Horizontal scalability enables resiliency • Multiple redundant managed nodes enable service availability in case of a node failure • Scale replication node to workload independent of DB node workload Scalability and availability Distributing computing architecture as enabler Horizontal Increase number of instances Vertical Increase resource size for node
- 10. Solution
- 11. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.11 Logical replication with EDB Replicate WHY HOW WHAT Protects data availability for mission critical applications to mitigate business risk EDB Replicate is built with Kafka to help eliminate single points of failure and provide high write- availability Provides a scalable, fault tolerant logical data replication for single- and multi-master replication use cases
- 12. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.12 How EDB Replicate works • Each node can publish and subscribe to changes • Logical decoding of WAL identifies SQL level changes published via Kafka • If a node fails another will take over • Failed database resumes from last sync point, in case of conflicts last update wins • Anti-affinity deployment pattern is used, i.e., EDB Replicate not co-located on database node Availability Zone 1 Availability Zone 3 Availability Zone 2 App App App EDB Replicate 1 EDB Replicate 2 EDB Replicate 3 EDB Replicate 3 takes over on failure at EDB Replicate 2 Cloud Service Provider
- 13. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.13 Software Architecture Apache Kafka Fault-tolerant distributed streaming replication Apache ZooKeeper Distributed configuration management Apache Avro Compressed data serialization and schema management Logical Decoding Logical change data capture component of Postgres database server Producer Streams from database to Kafka topic Consumer Streams from Kafka topic to database DB App CLI Logical Decoding EDB Replicate Consumer Producer ZooKeeperController Kafka Broker Topic Avro REST API
- 14. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.14 Apache Kafka Open source stream-processing platform • De facto infrastructure standard for capturing and publishing events • Used to integrate enterprise processes and workflows • Kafka collects the ‘digital exhaust’ of an enterprise for ML and continuous intelligence • Distributed architecture provides fault tolerance
- 15. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.15 Scalability and flexibility Feature Benefit • Horizontally scalable - add replication nodes (Kafka brokers) with no physical machine limits • Distributed - replication service is not co-located with the production database • Supports community PostgreSQL and EDB Postgres Advanced Server • Support different major versions of EDB Postgres Advanced Server or PostgreSQL in the same cluster • Increase processing horsepower to match replication workload without degrading database performance • Replication processes do not compete for the same resources as the PostgreSQL instance • Avoid vendor lock-in by using community PostgreSQL • Supports rolling upgrades with minimal/zero downtime
- 16. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.16 Fault tolerance and performance Feature Benefit • Fault-tolerant - resistant to replication node/machine failure within a cluster • Supports bidirectional replication; writes on any node are replicated to other PostgreSQL nodes • End-to-end data compression with encryption - Avro data storage format and Kafka data compression • No single point of failure. No need to retain WAL logs when the replication node fails • Enables complex architectures with multiple masters, manages conflicts for data consistency • Reduces message size sent between cluster nodes, thereby reducing data latency
- 17. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.17 Improves usability Feature Benefit • Add new DBs without seeding from production • Row-level filters • Multiple publications per DBs in same cluster • Email alerts based on thresholds, monitoring and centralized event tracking • High data availability without performance impact • Replication can be fine grained and integrate data from multiple DBs • Changes from multiple database table groupings can be replicated between DBs • Simplified maintenance of large clusters
- 18. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.18 • Encrypted database connection • SCRAM and certificate-based authentication • ACL-based access control • PostgreSQL 9.6, 10, 11, 12 • RHEL 7 & 8 on x86-64 as RPMs • Bundled Kafka with compression options Additional capabilities Environmental • CLI configuration management • Insert, delete and row level update-update conflicts managed • Multiple publications per database • Row level filtering • Cascading replication • Offline snapshots Functional
- 20. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.20 APAC EMEANA • Data availability for applications operating across multiple geographic regions • Updates must be to local region data only, ie, data model supports natural sharding • All regions have complete data set • Logistics applications are examples where data may be needed in multiple regions Geo-Distributed Minimize application latency for users M3M2M1 App App App Replication
- 21. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.21 • For multiple applications with common reference data • Application of record maintains data in master database • Replicas are read only for reference data used by applications • Inventory applications are an example where product catalog reference data is needed at a retail store Data Distribution Accurate and consistent shared reference data R2MR1 App1 App2 App3 Replication
- 22. Who is EDB?
- 23. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.23 EDB supercharges PostgreSQL Privately Held Major PostgreSQL community leader Over 5000 customers The largest provider of open-source PostgreSQL based software, services and support Founded in 2004 Over 10 years of consecutive quarterly subscription growth 500+ employees 16 offices worldwide
- 24. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.24 CONFIDENTIAL News: EDB Acquires 2ndQuadrant Now the largest dedicated provider of PostgreSQL products and solutions worldwide Brings together world’s top PostgreSQL experts, combining two best-in-class technical teams +
- 25. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.25 • Replication mitigates business risk by protecting data availability for mission critical applications • Modern PostgreSQL data replication must support the evolving multi-cloud needs of enterprises • EDB Replicate is designed for enterprise-class logical replication Summary
- 26. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.26 Learn More Other resources EDB Replicate General Availability late October Postgres Pulse EDB Youtube Channel Contact EDB info@edbpostgres.com