Best Practices & Lessons Learned from Deployment of PostgreSQL
1 like246 views
The document discusses best practices and lessons learned from the deployment of PostgreSQL, emphasizing the importance of understanding application landscapes and fostering collaboration for effective adoption. It highlights the increasing trend of organizations migrating to open source databases like PostgreSQL and provides recommendations for successful deployment, including automation and standardization. The document also outlines various deployment architectures and the significance of creating a supportive culture between developers and database administrators to enable effective use of PostgreSQL.
Best Practices & Lessons Learned from Deployment of PostgreSQL
- 1. Best practices & lessons learned from deployment of Postgres Marc Linster, CTO September 23, 2020
- 2. What is the challenge? • How to get the most mileage out of the best open source relational database • How do leaders do it? • Where do they start? • How do they drive the change?
- 3. Proven Practices ● Understand the application landscape and the fit for PostgreSQL ● Understand how to eat the elephant - one byte at a time ● Foster collaboration to drive adoption ● Use standardization to enable automation
- 4. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.4 Open Source wins in the RDBMS space By 2022, more than 70% of new in-house applications will be developed on an OSDBMS or OSDBMS-based dbPaaS, and 50% of existing proprietary RDBMS instances will have been converted or be in process of converting. By 2022, more than 60% of all global organizations will formalize their management of internal skills and standards related to OSDBMS to enable better use of open source in production. Gartner Group - State of the Open-Source DBMS Market, 2019
- 5. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.5 PostgreSQL won If you bet… you bet on PostgreSQL Most Loved DatabaseMost Commonly Used Database Postgres Popularity Source: Stack Overflow Developer Survey, 2019 Source: DB-Engines.com, 2020
- 6. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.6 Postgres is second most popular image in containers (after Nginx) Source: Datadog, Nov 2019, https://www.datadoghq.com/container-report/
- 7. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.7 100 TB+; read-only; relational +; multiple topics; rel. Integrity not relevant ; data scientist focus Hadoop, Cassandra Data volume and velocity Understand Where PostgreSQL Shines Data Lake Data Warehouse Data Mart ODS System of Record System of Engagement Edge Computing Mobile Computing 50TB+; read only; relational; star/cube/mat. view; periodic refresh; multiple topics; analyst focused 10TB+; read only; relational; star/cube/mat. view; periodic refresh; single topic; citizen analyst focus 10TB+; read only; relational only; normalized data model; concurrent refresh; citizen analyst focused 1-20TB; read/write; 90% relational; ACID; ERP/CRM, Line of business focused < 2TB; read/write; relational + doc + GIS; ACID or BASE; micro services; website and social media focus < 2TB; read/write; relational + doc + GIS; ACID or BASE; micro services; M2M, IoT focus < 1GB; read/write; relational + doc + GIS; single user; micro services IoT, mobile app focus Redshift, Snowflake, Teradata, Exadata, BigQuery Exadata, Oracle DB, SQL Server, Postgres Oracle, SQL Server, Postgres Oracle, SQL Server, Postgres MongoDB, MySQL, Redis, Postgres, MySQL, Postgres SQLite, Realm Innovation; cost; compatibility; no vendor lock in; Multi-model; innovation; cost; Available Evrywhr Hardware acceleration; MPP; S3 integration Hardware failover Native write scalability No single user footprint; no replication from/to mobile Multi-model; innovation; cost; Available Evrywhr Analytics capab.; compatibility; heterogeneous rep. Analytics capab.; compatibility; ease of integration. Description Key players PostgreSQL Strengths PostgreSQL Limitations
- 8. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.8 Create a Reasonable Plan Temptation: ● Go for major impact ● Address the worst legacy problem first Recipe for success: ● Tier your applications ● Develop reference architectures ● Understand your maturity ● Find the balance between feasible and meaningful
- 9. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.9 Define a Tiering Framework (sample framework) RPO: 24 hours RTO: 24 hours GRO: NA DB Failover time: NA An. maint. window: 24 hours SLA: 99.5% RPO: 1 min RTO: <1 min (Failover) 6 hours (PITR) GRO: NA DB Failover time: <1 min An. maint. window: 6 hours SLA: 99.99% RPO: No data loss RTO: <1 min (Failover) < 3 hours (PITR) GRO: 1 min DB Failover time: <1 min An. maint. window: 4 hours SLA: 99.999% RPO: No data loss RTO: <1 min (Failover) < 3 hours (PITR) GRO: 1 min DB Failover time: <1 min An. maint. window: 4 hours SLA: 99.999% RPO: No data loss RTO: <1 min (Failover) < 3 hours (PITR) GRO: 1 min DB Failover time: <1 min An. maint. window: 4 hours SLA: 99.999% Departmental Departmental Mission Critical Enterprise Mission Critical Enterprise Mission Critical High Volume Enterprise Mission Critical High Volume & High Scale < 1TB; < 500 TPS; < 100 users < 1TB; < 500 TPS; < 100 users < 5TB; < 5000 TPS; < 1000 users < 5TB; < 10,000 TPS; < 1000 users < 20TB; < 10,000 TPS; > 1000 users
- 10. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.10 Define Reference Architectures ● CPU, Memory, Storage ● Local, metro, geo-distributed ● Replicas for redundancy, read-scalability or both ● Streaming replication, logical replication or both? ● Connection pooling, query routing, or both? ● SLA, RPO, RTO, GRO
- 11. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.11 © Copyright EnterpriseDB Corporation, 2020. All rights reserved.11 Emerging • Getting started with PostgreSQL • Often for a single project • For test/dev apps Strategic • PostgreSQL considered a “standard” • Many production apps incl. tier 1 • Concerns for scalability, HA, and automation Expanding • Increasingly for production apps • Tools and automation important • Worries about database security Your PostgreSQL journey Value of PostgreSQL PostgreSQL Maturity Understand your Position on the Maturity Curve
- 12. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.12 Focus on Collaboration and Evangelizing Problem Evangelizing Enablement Successful open source adoption benefits from organizational change ● DBA teams and development teams are often at arms length ● “The infrastructure is ready, but nobody uses it” ● “We would build on Postgres, but the DBAs can’t handle it” ● Create a bottom-up innovation movement ● Informal discussions, lunch & learns, hackathons, meetups ● Create DevOps culture - break down the walls between developers and DBAs ● Make sure DBAs are ready to support production ● Consider Remote DBA Services ● Invest in automation to be responsive
- 13. Standardization and Automation • Open source == agility • Agility == speed and scale • There is no scalability without automation • Automation requires standardization
- 14. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.14 Core PostgreSQL Architectures Database servers • Single Node • Multi-node cluster with asynchronous replication • Multi-node cluster with synchronous replication
- 15. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.15 Single Node Property Description Recovery Time Objective Dependent on database size and backup strategy Recovery Point Objective Dependent on backup strategy Geographic Redundancy Objective N/A Target Availability 99.9%
- 16. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.16 Multi Node with Asynchronous Replication Property Description Recovery Time Objective < 60 seconds Recovery Point Objective Typically a few seconds or less during failover (un-replicated transactions may be lost). Dependent on backup strategy for full recovery. Geographic Redundancy Objective Dependent on node placement. Typically multi-availability zone. Target Availability 99.99%
- 17. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.17 Multi Node with Synchronous Replication Property Description Recovery Time Objective < 60 seconds Recovery Point Objective Typically a few seconds or less during failover, with no loss of committed transactions. Dependent on backup strategy for full recovery. Geographic Redundancy Objective Dependent on node placement. Typically multi-availability zone. Target Availability 99.99%
- 18. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.18 Postgres Deployment Automation Technology Decision Choices • Terraform • CloudFormation • Ansible • Chef • Puppet • Salt • Configuration Management Vs Provisioning • Procedural Vs Declarative • With local install or not • Mutable Infrastructure Vs Immutable Infrastructure • Agent Vs Agentless Factors to Consider
- 19. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.19 Deployment Tool Survey (June 2020)
- 20. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.20 EDB Postgres on Ansible Galaxy
- 21. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.21 Recommended Practices • Understand the application landscape and the fit for PostgreSQL • Understand how to eat the elephant - one byte at a time • Foster collaboration to drive adoption • Use standardization and automation to drive agility
- 22. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.22 Resources Thank You https://www.enterprisedb.com/white-papers ● Tuning Postgres on Windows and Linux https://www.enterprisedb.com/blog ● Reference architectures ● Automated deployment scripts (Ansible and Terraform) https://galaxy.ansible.com/edb_devops/edb_postgres ● Ansible Roles for deployment