What we've learned from running a PostgreSQL managed service on Kubernetes
0 likes93 views
The document discusses challenges and solutions for running a managed service using Kubernetes and PostgreSQL, particularly focusing on TimescaleDB, which is optimized for time-series data. Key issues include memory management leading to out-of-memory errors and maintaining uptime during PostgreSQL extension updates. The document also explores solutions such as the Oomguard library for memory allocation tracking and the use of a single primary with Patroni for high availability.
![Challenges
OOM killer
OOM causes abrupt shutdown of PostgreSQL
$ kubectl get pod tinyforkv01-an-0 -o json
jq '.spec.containers[0].resources'
{
"limits": {
"cpu": "4",
"memory": "1Gi"
},
"requests": {
"cpu": "4",
"memory": "1Gi"
}
}](https://image.slidesharecdn.com/oleksiikliukin-220516094727-f76f8829/85/What-we-ve-learned-from-running-a-PostgreSQL-managed-service-on-Kubernetes-11-320.jpg)
![Challenges
AWS bugs
Big encrypted EBS volumes
$ kubectl get statefulset l0c154j810-an | jq
'.spec.volumeClaimTemplates[0].spec'
{
"accessModes": [
"ReadWriteOnce"
],
"dataSource": {
"apiGroup": "snapshot.storage.k8s.io",
"kind": "VolumeSnapshot",
"name": "golden-snapshot--wsboilqtlr"
},
"resources": {
"requests": {
"storage": "2500Gi"
}
},
"storageClassName": "ebs-sc",
"volumeMode": "Filesystem"
}](https://image.slidesharecdn.com/oleksiikliukin-220516094727-f76f8829/85/What-we-ve-learned-from-running-a-PostgreSQL-managed-service-on-Kubernetes-25-320.jpg)
What we've learned from running a PostgreSQL managed service on Kubernetes
- 1. Running a managed service on Kubernetes and PostgreSQL What we learned at Timescale DoK Day Europe, May 16th, 2022 Oleksii Kliukin
- 2. TimescaleDB is PostgreSQL, supercharged for time-series data and analytics.
- 3. TimescaleDB is free and open PostgreSQL extension
- 4. Timescale Cloud is a hosted and fully managed TimescaleDB running on Kubernetes
- 8. Timescale cloud DB on Kubernetes / AWS
- 10. • Timescale continuous/real-time aggregates may require a lot of memory. Out of memory (OOM) when limits are set low is not uncommon. • OOM behavior assumed by PostgreSQL developers: • Linux OOM killer: SIGKILL a random Postgres process • A backend process is killed: disruption, restart of every connection • A postmaster is killed: unclean shutdown, in extreme cases to startup instance Challenges OOM killer OOM causes abrupt shutdown of PostgreSQL ERROR: out of memory on a request of 1024 bytes
- 11. Challenges OOM killer OOM causes abrupt shutdown of PostgreSQL $ kubectl get pod tinyforkv01-an-0 -o json jq '.spec.containers[0].resources' { "limits": { "cpu": "4", "memory": "1Gi" }, "requests": { "cpu": "4", "memory": "1Gi" } }
- 12. Challenges OOM killer OOM causes abrupt shutdown of PostgreSQL * Image source: user MesserWoland, https://en.m.wikipedia.org/wiki/File:The_death.svg *
- 13. • Regular PostgreSQL: set memory overcommit, enable swap • vm.overcommit_memory = 2 • Can’t set it individually per container • A node typically runs some pods (eg. daemonsets for logging) incompatible with this setting https://github.com/kubernetes/kubernetes/issues/90973 Challenges OOM killer OOM causes abrupt shutdown of PostgreSQL
- 14. • Solution: OOMGuard library collects statistics on the memory usage, overriding malloc • Use LD_PRELOAD_LIBRARY to install it for Postgres processes • Can just report statics, or actually block allocations going above the predefined threshold, emulating regular malloc behavior • OOM_GUARD_LIMIT threshold is derived from the container memory limit, accounting for shared_buffers and OS overhead. Challenges OOM killer OOM causes abrupt shutdown of PostgreSQL
- 15. • PostgreSQL: provide memory allocation hooks to do internal accounting and deny allocations via extensions. • Linux/Kubernetes: configure oom_adj_score and vm_overcommit per cgroup on the Linux/Kubernetes layer. • Improved debugging experience (locating debug symbols from the container when running perf or gdb on the host) Challenges OOM killer Wishlist
- 16. • New versions of Timescale extension are released regularly • A new timescale-docker-ha image is built once the extension is released • We want to deliver latest extension (but not necessary auto- upgrade) to our customers immediately • Changing pod’s docker image requires a pod restart • Planned customer downtime may only happen during maintenance window, only a few times a year Challenges Fewer downtimes Extension updates require a pod bounce
- 17. Challenges Fewer downtimes Extension updates require a pod bounce
- 18. • Solution: hot-forge • A binary inside the container to fetch pre-packaged bundles and put them in the container • The bundles are delivered using a postgres connection (COPY TO PROGRAM) • The bundles are written to a persistent volume and linked to a container filesystem • Mostly adding new data (although can potentially replace/delete existing files in the container) Challenges Fewer downtimes Extension updates require a pod bounce
- 19. • Allow bouncing of individual containers in the pod and changing the docker image • Support “mutable” area inside the pod to deliver updates. Challenges Fewer downtimes Wishlist
- 20. • Etcd is a core of the Kubernetes cluster • Consists of multiple nodes (we run 5) - should be resilient? • Can degrade on master node updates • Performance issues (EBS burst balance, too many objects) • Patroni dependency (no Kubernetes API - instances are read- only) Challenges Operating Etcd Etcd is a 5-nodes single point of failure Image source: Andy Langager, https://www.flickr.com/photos/andylangager/8631388149
- 21. • Solution: no silver bullet • Many small clusters in each region instead of a single big one • Etcd performance monitoring • Fire drills on ephemeral clusters • Solution: Patroni experimental static_primary mode: • Enforce single primary by rejecting connections from other nodes • Do not demote when Kubernetes API is not available Challenges Operating Etcd Etcd is a 5-nodes single point of failure
- 22. • Some operational instructions when Etcd is down • Better observability inside Etcd • Patroni “isolated” mode scalable to any number of pods Challenges Operating Etcd Wishlist
- 23. • New (1TB+) encrypted EBS volumes show an existing partition marker (Atari partition) • Kubernetes refuses to format them • Pod is stuck at startup Challenges AWS bugs Encrypted EBS volumes Image source: Bill Bertram, https://commons.wikimedia.org/wiki/File:Atari_1040STf.jpg
- 24. • Solution: create a small 1GB encrypted volume • Snapshot it into a “golden snapshot” • Create new encrypted volumes from the golden snapshot • Need to resize the filesystem in the init container (as per Kubernetes 1.19) • Recent fix by AWS: https://github.com/kubernetes/kubernetes/ issues/86064 Challenges AWS bugs Big encrypted EBS volumes
- 25. Challenges AWS bugs Big encrypted EBS volumes $ kubectl get statefulset l0c154j810-an | jq '.spec.volumeClaimTemplates[0].spec' { "accessModes": [ "ReadWriteOnce" ], "dataSource": { "apiGroup": "snapshot.storage.k8s.io", "kind": "VolumeSnapshot", "name": "golden-snapshot--wsboilqtlr" }, "resources": { "requests": { "storage": "2500Gi" } }, "storageClassName": "ebs-sc", "volumeMode": "Filesystem" }
- 26. Challenges AWS bugs • Fewer bugs :-) • Improved support for VolumeSnapshots, e.g. provisioning volumes across namespaces, resizing a filesystem when provisioning from a snapshot Wishlist
- 27. • AWS EBS and other PersistentVolume implementations only allow volume size increments. • A volume autoscaler (Timescale service) may decide to increase the volume upon a data ingestion • When data is subsequently compressed the customer doesn’t need to pay for a bigger volume Challenges Volume resize Volume size can only be increased, not decreased
- 28. • Solution: provide a functionality to fork a service • A fork is a clone of a service with possibly different CPU and storage specs • A fork is implemented by restoring another instance from the backup of the original one, taken from S3 Challenges Volume resize Volume size can only be increased, not decreased
- 29. • Native volume downsize • Kubernetes support, possibly with custom checks from K8s to determine this is possible. • Support for volume resizing in a statefulset Challenges Volume resize Wishlist
- 31. Feedback and Questions: oleksii@timescale.com Twitter: @hintbits