Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)
1 like3,842 views
The document outlines Zalando's implementation of a highly available PostgreSQL cluster using Spilo and Patroni, managing over 150 production databases and addressing common database failures. It discusses the use of AWS for cloud services, microservices architecture, and the responsibilities of development teams regarding database infrastructure and failover management. Key features include automated failover, streaming replication, and the use of etcd for distributed consensus and state management.
Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)
- 2. Zalando • 15 EU countries • 3 fulfilment centers • 15+ million active customers • 2.2 billion € revenue 2014
- 6. Our databases • >150 production Postgresql databases • >13.5 TB data • >5 TB biggest DB • 400-1000+ write tps • >2 DB failures/month
- 10. Purpose
- 11. Autonomy
- 12. Mastery
- 13. Cloud • 2013: ZCloud • 2014: project Pequod • 2015: Let’s just use AWS…
- 14. Amazon 3-letter words • AWS - amazon web services • EC2 - elastic compute cloud • ELB - elastic load balancer • RDS - relational DB service
- 15. AWS • One account per team • Microservices • REST/OAuth2 • Deployment with Docker
- 16. Autonomous teams on AWS REST INTERNET
- 17. Autonomous teams • Team decides which product to build • … and which technologies to use • REST/OAuth2 mandatory • Team is responsible for its infrastructure
- 18. Databases? • Developers should take care of infrastructure • ..including production databases • On AWS!
- 19. Isn’t it dangerous? DBAs running with scissors, by Gavin M. Roy: https://www.flickr.com/photos/gavinmroy/4638958958
- 22. Autofailover tasks • Detect the master failure • Elect a new master • Redirect clients
- 23. Autofailover issues • Discarded writes • Split-brain • False positives
- 24. RDS? • Support for PostgreSQL • Automatic failover • Most extensions • Automatic backups
- 25. RDS? • Vendor lock • No superuser • No untrusted languages • No logical decoding plugins • Rather expensive
- 26. EC2 + Linux HA • Complex setup • Lots of manual steps (i.e. new replica creation)
- 27. Spilo (!"#$%)
- 28. Spilo does • Rapid deployment of PostgreSQL on AWS EC2 instances • Streaming replication with auto-failover
- 29. Spilo on AWS Spilo MASTER Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
- 30. Failover Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
- 31. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update NEW SPILO STARTS…
- 32. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update Spilo REPLICA
- 33. What is Spilo? c Patroni MASTER c Patroni REPLICA c Patroni REPLICA Auto-scaling group Auto-scaling group
- 34. Patroni ("&'(%)#) • Handles new replicas and failover • Based on ideas and code of the Compose Governor • Open-source
- 36. Distributed configuration systems • Fault tolerant • Reliably store small amounts of strongly-consistent data between distributed nodes • Good for storing the PostgreSQL cluster state
- 37. Distributed consensus LEADER CLIENT CLIENT CLIENT
- 38. Distributed consensus LEADER CLIENT CLIENT CLIENT
- 39. Cluster state in etcd $ etcdctl ls --recursive /service /service/batman /service/batman/optime /service/batman/optime/leader /service/batman/members /service/batman/members/postgresql0 /service/batman/members/postgresql1 /service/batman/initialize /service/batman/leader
- 40. Leader key $ etcdctl get /service/batman/leader postgresql0 • Points to the member key • Has a TTL, autoexpires • Acts as an exclusive lock • Only the leader can become the master
- 41. Leader TTL $ http http://127.0.0.1:2379/v2/keys/service/batman/ leader … { "action": "get", "node": { "createdIndex": 48723, "expiration": "2015-10-23T14:51:49.686506977Z", "key": "/service/batman/leader", "modifiedIndex": 49037, "ttl": 27, "value": "postgresql0" } }
- 42. Member key $ etcdctl get /service/batman/members/ postgresql0 {“role":"master", “state”:"running", “conn_url”:"postgres://replicator:rep- pass@127.0.0.1:5432/postgres", “api_url”:"http://127.0.0.1:8008/ patroni", "xlog_location":67108960}
- 43. Connection and API URL c Patroni c Patroni API URL (check health during promotion) MASTER REPLICA CONNECTION URL MASTER LB REPLICA LB CONNECTION URL
- 44. Initialize key $ etcdctl get /service/batman/initialize 6208852353820383446 • PostgreSQL cluster system ID • Created by the first node that joins the cluster • Nodes with different system ID are not allowed to join
- 45. Patroni modules ETCD ZOOKEEPER ABSTRACT DCS PostgreSQL REST API High availability Asynchronous executor Callbacks
- 47. From Governor to Patroni Governor Patroni
- 48. Location of etcd: original c Governor c Governor c Governor
- 49. Replace etcd with proxy c Governor c Governor c Governor Proxy Proxy Proxy
- 50. Embed etcd client in Patroni c Patroni c Patroni c Patroni
- 51. Patroni improvements • Robust exception handling • Run long-running tasks (i.e. base backup in a separate thread) • ETCD + Zookeeper • Rest API
- 52. Patroni improvements • Configurable replica imaging • Support for pg_rewind
- 53. Patroni improvements • Manual failover • Initialize from external cluster • Attach to already running PostgreSQL nodes • Tags (i.e. nofailover)
- 54. What you should monitor • replication lag • unhealthy member • no leader • etcd/ Zookeeper
- 55. Thank you! • Spilo: github.com/zalando/spilo spilo.readthedocs.org • Patroni: github.com/zalando/patroni patroni.readthedocs.org • Feedback: @alexeyklyukin