PGConf APAC 2018 - PostgreSQL HA with Pgpool-II and whats been happening in Pgpool-II world lately....
1 like712 views
The document discusses the implementation of high availability (HA) in PostgreSQL through the use of Pgpool-II and the improvements made over time, including the introduction of a watchdog for failure management. It emphasizes the importance of treating database servers as a herd to ensure seamless scalability and performance, while also addressing challenges such as single points of failure. Recent enhancements and features are highlighted, showcasing Pgpool-II's evolving reliability and efficiency in handling database management.
PGConf APAC 2018 - PostgreSQL HA with Pgpool-II and whats been happening in Pgpool-II world lately....
- 1. © 2016 EDB. All rights reserved. 1 PostgreSQL HA with Pgpool-II and whats been happening in Pgpool-II world lately...
- 2. © 2018 EDB. All rights reserved. 2 • Muhammad Usama • Database Architect with EnterpriseDB • Pgpool-II developer and committer • Watchdog overhauling • New PCP system for Pgpool-II • Authentication method support muhammad.usama@enterprisedb.com About Me
- 3. © 2018 EDB. All rights reserved. 3 • We treat our servers like pets (e.g. Sunshine the cow). If Sunshine gets sick, It feels like the end of the world and everything stops there Pets Vs Cattle Herd • We treat our servers like cattle herd (C01, C02, … C99). If some cattle gets sick, there is system in place to isolate it from the herd and things go on.
- 4. © 2018 EDB. All rights reserved. 4 • Sunshine the cow is unique and indispensable • Sunshine the cow is hand fed and hand crafted Pets Vs Cattle Herd • Cannot handle failure • Herd of similar cattle (C01,C02..) • No cattle is special and indispensable • Can handle failures
- 5. © 2018 EDB. All rights reserved. 5 • Herd approach can be used on database servers to achieve performance scalability and high availability. How? • Make a homogeneous copies of database servers (replicated databases) • Have a system to manage the herd and failures. Using the herd approach for HA in database
- 6. © 2018 EDB. All rights reserved. 6 • Herd needs a leader (Primary server). • A system is required to elect new leader if the current leader becomes unavailable (Primary node failure). • Needs a mechanism to make herd follow the new leader. • We need a system to seamlessly retire the sick nodes (Standby node failure). • A system to add new nodes without effecting the service. • Require a system to make the whole herd work in collaboration to efficiently utilise the resources (Load balancing) Challenges in the herd approach
- 7. © 2018 EDB. All rights reserved. 7 The Solution
- 8. © 2018 EDB. All rights reserved. 8 • Cluster management tool dedicated for PostgreSQL • Rich in features • Connection pooling • Load balancing • Automatic failover • Query caching • Watchdog (High availability) • OSS project, BSD License What is Pgpool-II?
- 9. © 2018 EDB. All rights reserved. 9 • Pgpool-II make the cluster appears as a single PostgreSQL instance • All standard PostgreSQL clients work seamlessly • Automatic failover • Provides flexibility and control to select the primary node when old primary fails • Load balancing Solving the problem using Pgpool-II
- 10. © 2018 EDB. All rights reserved. 10 Basic idea of Pgpool-II
- 11. © 2018 EDB. All rights reserved. 11 • If primary goes down, one of the standby is promoted to the new primary Primary server fails • Other standby nodes are made to follow new primary
- 12. © 2018 EDB. All rights reserved. 12 • If the standby goes down, It is simply taken off from the cluster Standby server fails • Users continue to use the service without disruption
- 13. © 2018 EDB. All rights reserved. 13 • Mechanism in Pgpool-II to detach problematic nodes • Reconfigure the standby nodes to follow new primary node • Automatically triggers • When health check monitor the node failure • Reading/writing failure to PostgreSQL backend (fail_over_on_backend_error is on) • By remote Pgpool-II node (Watchdog) Automatic failover
- 14. © 2018 EDB. All rights reserved. 14 • Using Pgpool-II with failover and health check ensures the service availability when PostgreSQL node fails • But there is still a problem Does that solves the problem?
- 15. © 2018 EDB. All rights reserved. 15 • Pgpool-II node failure can cause service unavailability even when all db servers are healthy What if Pgpool-II fails • Pgpool-II becomes the single point of failure (SPOF)
- 16. © 2018 EDB. All rights reserved. 16 Watchdog
- 17. © 2018 EDB. All rights reserved. 17 • A sub process of Pgpool-II to handle failures • Life checking of Pgpool-II service • Mutual monitoring of Pgpool-II nodes in the cluster • Leader election to select best master node • Virtual-IP control • Ensuring same view of PostgreSQL backend states across all Pgpool-II nodes • Distributed failover management What is Watchdog?
- 18. © 2018 EDB. All rights reserved. 18 Pgpool-II with watchdog
- 19. © 2018 EDB. All rights reserved. 19 Pgpool-II node failure
- 20. © 2018 EDB. All rights reserved. 20 • Pgpool-II with Watchdog Recipe for PostgreSQL HA
- 21. © 2018 EDB. All rights reserved. 21 whats been happening in Pgpool-II world lately...
- 22. © 2018 EDB. All rights reserved. 22 • Pgpool-II is becoming more reliable • Watchdog is getting smarter • PCP command enhancements • Failover is getting better • Performance improvements • Many new features Whats been happening in Pgpool-II lately..
- 23. © 2018 EDB. All rights reserved. 23 • Watchdog was very basic until Pgpool-II 3.5 • So many complaints around split- brain syndrome • Scalability and maintainability were big issues • Rewritten in Pgpool-II 3.5 Watchdog is getting smarter
- 24. © 2018 EDB. All rights reserved. 24 • Ensures the quorum to avoid spilt- brain syndrome during leader election • Actively look out for split-brain syndrome No more split-brain syndrome
- 25. © 2018 EDB. All rights reserved. 25 • Watchdog priority (wd_priority config parameter) can be assigned to the Pgpool-II node • Weighs various node attributes (number of connections, uptime, age, priority) Better voting and leader election mechanism
- 26. © 2018 EDB. All rights reserved. 26 • Allow third-party integrations • Expose interfaces for external health check systems to integrate with watchdog • Extendable and maintainable Watchdog now uses sockets and JSON data format for IPC
- 27. © 2018 EDB. All rights reserved. 27 • Pgpool-II nodes collaborate to validate backend node failure • Failover is performed democratically using quorum • Tolerate network glitches to avoid unnecessary failover • The behaviour is configurable • failover_when_quorum_exists • failover_require_consensus • enable_multiple_failover_requests_from_node Watchdog side enhancements in the failover
- 28. © 2018 EDB. All rights reserved. 28 • Large SELECT query performance improvement • Improvements in extended query performance • Thundering herd problem fix Performance Updates
- 29. © 2018 EDB. All rights reserved. 29 • pool_hba enhancements • Advancement of backend node health checking • Support AWS Aurora • New PGPOOL SET command to change Pgpool-II configurations on the fly • Minimal session disconnections with improved failover mechanism • Pg_terminate_backend handling • Documentation improvement Some recent notable enhancements in Pgpool-II
- 30. © 2018 EDB. All rights reserved. 30 • Support of additional authentication methods • SCRAM • CERT Whats in the pipeline
- 31. © 2018 EDB. All rights reserved. 31 https://pgpool.net/mediawiki/index.php/pgpool-II_3.8_development Whats in the pipeline
- 32. © 2018 EDB. All rights reserved. 32 Thank you!