Operating PostgreSQL at Scale with Kubernetes
6 likes2,774 views
The document discusses the implementation and management of PostgreSQL databases at scale using Kubernetes, highlighting the advantages and challenges of using containers. It covers the use of Crunchy Data's container suite for deployment, monitoring, and disaster recovery, and illustrates how operators can simplify administration tasks. The conclusion emphasizes the potential of PostgreSQL combined with containers and Kubernetes for creating efficient development environments and automated management solutions.
Operating PostgreSQL at Scale with Kubernetes
- 1. Operating PostgreSQL at Scale With Kubernetes JONATHAN S. KATZ MARCH 7, 2019 SCALE17X
- 2. • Director of Communications, Crunchy Data • Previously: Engineering leadership in startups • Longtime PostgreSQL community contributor • Advocacy & various committees for PGDG • @postgresql + .org content • Director, PgUS • Conference organization + speaking • @jkatz05 About Me 2
- 3. About Crunchy Data 3 Market Leading Data Security • Crunchy Certified PostgreSQL is open source and Common Criteria EAL 2+ Certified, with essential security enhancements for enterprise deployment • Author of the DISA Secure Technology Implementation Guide for PostgreSQL and co-author of CIS PostgreSQL Benchmark. Move ATO from weeks to days! Cloud Ready Data Management • Open source, Kubernetes-based solutions proven to scale to 1000s of database instances • Cloud-agnostic technology provide flexibility on how to deploy databases to public clouds, private clouds, or on-premise technology Leader in Open Source Enterprise PostgreSQL • Developer of essential open source tools for high availability, disaster recovery, and and monitoring for PostgreSQL • Leading contributor and sponsor of features that enhance stability, security, and performance of PostgreSQL
- 4. • Containers + PostgreSQL • Setting up PostgreSQL with Containers • Operating PostgreSQL at Scale With Kubernetes • Look Ahead: Trends in the Container World Outline 4
- 5. • Containers provide several advantages to running PostgreSQL: • Setup & distribution for developer environments • Ease of packaging extensions & minor upgrades • Separate out secondary applications (monitoring, administration) • Automation and scale for provisioning and creating replicas, backups Containers & PostgreSQL 5
- 6. • Containers also introduce several challenges: • Administrator needs to understand and select appropriate storage options • Configuration for individual database specifications and user access • Managing 100s - 1000s of containers requires appropriate orchestration (more on that later) • Still a database within the container; standard DBA tuning applies • However, these are challenges you will find in most database environments Containers & PostgreSQL 6
- 7. • We will use the Crunchy Container Suite • PostgreSQL (+ PostGIS): our favorite database; option to add our favorite geospatial extension • pgpool + pgbouncer: connection pooling, load balancing • pgBackRest: terabyte-scale disaster recovery management • Monitoring: pgmonitor • pgadmin4: UX-driven management • Open source! • Apache 2.0 license • Support for Docker 1.12+, Kubernetes 1.5+ • Actively maintained and updated Getting Started With Containers & PostgreSQL 7 https://github.com/CrunchyData/crunchy-containers
- 8. Getting Started With Containers & PostgreSQL 8
- 9. Demo: Creating & Working With Containerized PostgreSQL 9 mkdir postgres && cd postgres docker volume create --driver local --name=pgvolume docker network create --driver bridge pgnetwork cat << EOF > pg-env.list PG_MODE=primary PG_PRIMARY_USER=postgres PG_PRIMARY_PASSWORD=password PG_DATABASE=whales PG_USER=jkatz PG_PASSWORD=password PG_ROOT_PASSWORD=password PG_PRIMARY_PORT=5432 PG_LOCALE=en_US.utf8 PGMONITOR_PASSWORD=monitorpassword EOF docker run --publish 5432:5432 --volume=pgvolume:/pgdata --env-file=pg-env.list --name="postgres" --hostname="postgres" --network="pgnetwork" --detach crunchydata/crunchy-postgres:centos7-11.2-2.3.1
- 10. Demo: Adding in pgadmin4 10 docker volume create --driver local --name=pga4volume cat << EOF > pgadmin4-env.list PGADMIN_SETUP_EMAIL=jonathan.katz@crunchydata.com PGADMIN_SETUP_PASSWORD=securepassword SERVER_PORT=5050 EOF docker run --publish 5050:5050 --volume=pga4volume:/var/lib/pgadmin --env-file=pgadmin4-env.list --name="pgadmin4" --hostname="pgadmin4" --network="pgnetwork" --detach crunchydata/crunchy-pgadmin4:centos7-11.2-2.3.1
- 11. Demo: Adding Monitoring 11 cat << EOF > collect-env.list DATA_SOURCE_NAME=postgresql://ccp_monitoring:monitorpassword@postgres:5432/postgres?sslmode=disable EOF docker run --env-file=collect-env.list --network=pgnetwork --name=collect --hostname=collect --detach crunchydata/crunchy-collect:centos7-11.2-2.3.1 mkdir prometheus cat << EOF > prometheus-env.list COLLECT_HOST=collect SCRAPE_INTERVAL=5s SCRAPE_TIMEOUT=5s EOF docker run --publish 9090:9090 --env-file=prometheus-env.list --volume `pwd`/prometheus:/data --network=pgnetwork --name=prometheus --hostname=prometheus --detach crunchydata/crunchy-prometheus:centos7-11.2-2.3.1 mkdir grafana cat << EOF > grafana-env.list ADMIN_USER=jkatz ADMIN_PASS=password PROM_HOST=prometheus PROM_PORT=9090 EOF docker run --publish 3000:3000 --env-file=grafana-env.list --volume `pwd`/grafana:/data --network=pgnetwork --name=grafana --hostname=grafana --detach crunchydata/crunchy-grafana:centos7-11.2-2.3.1 1. Set up the metric collector 2. Set up prometheus to store metrics 3. Set up grafana to visualize
- 12. Running PostgreSQL on Kubernetes. At Scale.
- 13. • Value of Kubernetes increases exponentially as number of containers increases • Running databases on Kubernetes requires more specialized knowledge than running non- stateful applications • What happens to your data after a pod goes down? When to Use Kubernetes with PostgreSQL 13
- 14. • PostgreSQL Operator GA: March, 2017 • Allows an administrator to run PostgreSQL-specific commands to manage database clusters, including: • Creating / Deleting a cluster (your own DBaaS) • Scaling up / down replicas • High-Availability • Apply user policies to PostgreSQL instances • Managing backup intervals and policies • Define what container resources to use (RAM, CPU, etc.) • Upgrade management • Smart pod deployments to nodes • REST API Crunchy PostgreSQL Operator 14 https://github.com/CrunchyData/postgres-operator
- 15. • Utilizes Kubernetes Deployments: • Flexibility in storage classes • Flexibility in operating environments • Node affinity • Resource (CPU, RAM) configurations • Flexibility in database version runtimes Crunchy PostgreSQL Operator: Architecture 15
- 16. • Automation: Complex, multi-step DBA tasks reduced to one-line commands • Standardization: Many customizations, same workflow • Ease-of-Use: Simple CLI • Scale • Provision & manage clusters quickly amongst thousands of instances • Load balancing, disaster recovery, security policies, deployment specifications • Security: Sandboxed environments, RBAC, mass grant/revoke policies Why Use An Operator With PostgreSQL? 16
- 17. Why Use An Operator With PostgreSQL? 17
- 18. Demo: Provisioning a Cluster 18 pgo create cluster --autofail --pgbackrest --metrics --replica-count 1 scale17x pgo show cluster scale17x
- 19. Demo: Creating a User; Connectivity; Utilization 19 pgo create user jkatz scale17x --password password --managed --selector=name=scale17x pgo test scale17x pgo df scale17x
- 20. Demo: Running Some Tests; Utilization 20 # get the service forward command # run some pgbench pgbench -i -s 1 -h localhost -p 5434 userdb pgbench -c 2 -j 1 -t 128 --progress=1 -h localhost -p 5434 userdb pgbench -c 2 -j 1 -t 128 -S --progress=1 -h localhost -p 5434 userdb # Coming in 4.0: pgo benchmark! pgo df scale17x
- 21. Demo: Labels; Here is Where We Scale! 21 # labels pgo label scale17x --label=project=current pgo create cluster scale18x --labels project=future pgo create cluster scale19x --labels project=future pgo show cluster --selector=project=future pgo create user jkatz --password password --managed --selector=project=future pgo delete user jkatz --selector=project=future
- 22. Demo: High-Availability and Horizontal Scaling 22 # It's elastic! pgo scale scale17x --replica-count=1 # Run some queries on the replica # HA pgo failover scale17x --query pgo failover scale17x --autofail-replace-replica true --target <pod> pgo test scale17x
- 23. Demo: Setting Backup Policies 23 # backup policy pgo create schedule scale17x --schedule="0 0 * * *" --schedule-type=pgbackrest --pgbackrest-backup-type=full pgo create schedule scale17x --schedule="0 6,12,18 * * *" --schedule-type=pgbackrest --pgbackrest-backup-type=diff pgo show schedule scale17x
- 24. Demo: Disaster Strikes! 24 pgo backup scale17x --backup-type=pgbackrest # log in, do some stuff # oh no! restore # can choose to do point-in-time-recovery # pgo restore scale17x --backup-type=pgbackrest --pitr-target="2019-03-07 17:44:00" - backup-opts="--type=time" # or choose to back up up until the last archive # pgo restore scale17x --backup-type=pgbackrest
- 25. PostgreSQL & Containers: Looking Ahead
- 26. • Containers are no longer "new" - orchestration technologies have matured • Debate with containers + databases: storage & management • No different than virtual machines + databases • Databases are still databases: need expertise to manage • Stateful Sets vs. Deployments • Federation v2 API opens up new possibilities for high-availability • Database deployment automation flexibility • Deploy your architecture to any number of clouds • Monitoring: A new frontier Containerized PostgreSQL: Looking Ahead 26
- 27. Conclusion 27 • PostgreSQL + Containers + Kubernetes gives you: • Easy-to-setup development environments • Your own production database-as-a-service • Tools to automate management of over 1000s of instances in short-order