PostgreSQL Monitoring using modern software stacks

https://tech.showmax.com@ShowmaxDevs
PostgreSQL Monitoring
Using modern software stacks
Roman Fišer

About me
● Roman Fišer
● Head of Infrastructure
● Showmax Engineering
● roman.fiser@showmax.com

● VOD (Video On Demand) service
● Focusing on clients in Africa
● Engineering in Prague
● Based on Open source technologies
What is Showmax?

Open Source

PostgreSQL
● Cluster management handled by
Patroni
● High-availability
■ Automatic master/slave election
■ Auto-failover
● Streaming replication
● Written in python & Open Source
■ Multiple patches to upstream
● Secrets management w Vault
● Barman for Backups

PostgreSQL @ Showmax
● Backend - data store for Showmax business entities
● Microservice architecture, each DB has RESTful microservice
● CMS
● Stores CMS in PostgreSQL (then denormalized to Elastic)
● Cache invalidations
● Analytics DWH
● Copy over from all databases
● Events digestion

What type of metrics are important?
● Four SRE Golden Signals
● Latency
■ The time it takes to service a request
● Traffic
■ A measure of how much demand is being placed on your system
● Error
■ The rate of requests that fail
● Saturations
■ How "full" your service is

Latency
● pg_stat_statements
● Average call time
● Maximum call time
● Replication delay

Traffic
● Pg_stat_statements
● Calls
● Returned rows
● Network traffic
● System IO Statistics (IOPS, traffic)

Errors
● Rollback / Commit ratio
● Deadlocks

Saturation
● num_backends / max_connections
● High IO utilization (iowait, await)
● High CPU utilization
● Checkpoints
● Tempfile usage
● Disk usage (free space, inodes)

Other important metrics
● Long running idle in transaction
● Blocked autovacuum
● Error events in PostgreSQL log
● Server crashes
● I/O Errors
● Data corruption
● Index corruption

Tools - Prometheus Stack
● Prometheus
● Prometheus is a time-series database. Suitable for white-box monitoring
● Alert Manager
● Part of the Prometheus project. Used for Alerting.
● Exporters
● Patroni_exporter, Postgres_exporter - Exports PostgreSQL metrics
● Node_exporter - Expose OS metrics
● Grafana
● Web frontend for Prometheus data

Prometheus vs Nagios/Icinga
Nagios/Icinga
● Focus on black-box monitoring
● Checks usually complicated bash scripts
● Can’t base alerts on relations between different metric types
Prometheus
● Promotes white-box monitoring
● High-performance TSDB
● Cloud-native ready with multiple service discovery providers
● Standardized interface for exporters

● Monitoring based on metrics with metadata (CPU,
RAM, disk IO, disk utilization, etc.)
● Custom labels for metrics
● Functions to filter, change, remove …. metadata
while fetching them
● Multiple exporters - expose data via HTTP API
● Effective data fetching:
● Based on intervals measured in seconds
● Million of data points
● Notifications can be reported via Email, Slack, etc.
Prometheus
Prometheus
https://prometheus.io

Prometheus Pipeline

Prometheus
PSQL exporter
Postgresql
GET /metrics
(HTTP query)
SQL query
(dedicated user)
Plain text
response
How does it work?

PromQL

Alert Manager Rules - Latency
- alert: PgSQLSlowLoginQuery
expr: avg_over_time(pg_stat_statements_mean_time_seconds{datname="cms", queryid="3985044216"}[1m])
> 100
for: 5m
labels:
severity: critical
team: ops
annotations:
summary: "Slow login query (instance {{ $labels.instance }})"
description: |
Login queries are too slow (> 30s). Average is {{ $value }}.
Check the PostgreSQL instance {{ $labels.instance }}
runbook: pgsql@pgsqlslowloginquery
title: PgSQLSlowLoginQuery

Alert Manager Rules - Latency
- alert: PgSQLReplicationLagIsTooBig
expr: pg_replication_lag{instance!~"^(ba-patroni|analytics-patroni).*"} > 300 and pg_is_in_recovery == 1
for: 15m
labels:
severity: critical
team: ops
annotations:
description:
Replication lag on PostgreSQL Slave {{ $labels.instance }} is
{{ humanizeDuration $value }}. If lag is too big, it might be
impossible for Slave to recover, it might not be considered as a new
Patroni leader, or data loss could occur should such Slave be chosen as next
leader.
summary: PostgreSQL Slave Replication lag is too big.
runbook: pgsql#pgsqlreplicationlagistoobig
title: PgSQLReplicationLagIsTooBig

Alert Manager Rules - Traffic
- alert: PgSQLCommitRateTooLow
expr: |
rate(pg_stat_database_xact_commit{datname="oauth", sm_env="prod"}[5m]) < 200
for: 5m
labels:
severity: warn
team: ops
annotations:
description: |
Commit Rate {{$labels.instance}} for database {{$labels.datname}}
is {{$value}} which is suspiciously low.
runbook: pgsql#pgsqlcommitrateislow
title: PgSQLCommitRateTooLow

Alert Manager Rules - Saturation
- alert: PgSQLNumberOfConnectionsHigh
expr: (100 * (sum(pg_stat_database_numbackends) by (instance, job) / pg_settings_max_connections)) > 90
for: 10m
labels:
severity: critical
team: ops
annotations:
description:
Number of active/open connections to PostgreSQL on {{ $labels.instance }}
is {{ $value }}. It's possible PostgreSQL won't be able to accept any
new connections.
summary: Number of active connections to Postgresql too high.
runbook: pgsql#pgsqlnumberofconnectionshigh
title: PgSQLNumberOfConnectionsHigh

Alert Manager Rules - Errors
- alert: PgSQLRollbackRateTooHigh
expr: |
rate(pg_stat_database_xact_rollback{datname="oauth"}[5m])
/ ON(instance, datname)
rate(pg_stat_database_xact_commit{datname="oauth"}[5m])
> 0.05
for: 5m
labels:
severity: warn
team: ops
annotations:
description: |
Ratio of transactions being aborted compared to committed is
{{$value | printf "%.2f" }} on {{$labels.instance}}
runbook: pgsql@pgsqlrollbackrateishigh
title: PgSQLRollbackRateTooHigh

- alert: PgSQLDeadLocks
expr: rate(pg_stat_database_deadlocks{datname!~"template.*|postgres"}[1m]) > 0
for: 5m
labels:
severity: warn
team: ops
annotations:
description: |
Deadlocks has been detected on PostgreSQL {{ $labels.instance }}.
Number of deadlocks: {{ $value }}
summary: "Dead locks (instance {{ $labels.instance }})"
runbook: pgsql#pgsqldeadlocksdetected
title: PgSQLDeadLocks

- alert: PgSQLTableNotVaccumed
expr: time() - pg_stat_user_tables_last_autovacuum{datname="oauth", sm_env="prod"} > 60 * 60 * 24
for: 5m
labels:
severity: warn
team: ops
annotations:
summary: "Table not vaccumed (instance {{ $labels.instance }})"
description: |
Table has not been vaccum for 24 hours {{ $labels.relname }}
(vacuumed before {{ humanizeDuration $value }}). There may be not enough vacuum workers.
runbook: pgsql#pgsqltablenotvacuumed
title: PgSQLTableNotVaccumed

Grafana

Prometheus caveats
● Beware of labels with high cardinality
○ Significant performance penalty
○ It is not possible to remove labels from DB

Tools - ELK Stack
● Elastic
● Search and analytics engine
● RabbitMQ
● OSS Message broker. Used for log messages delivery
● Logstash
● Server‑side data processing pipeline
● Kibana
● Kibana lets users visualize data with charts and graphs in Elasticsearch.

Logging Pipeline

Kibana

● Simple framework to alert anomalies, spikes, or
other patterns of interest from data in
Elasticsearch
● Two types of components:
● Rule types (frequency, spike, flatline, etc.)
● Alert types (email, slack, OpsGenie, etc.)
● Alerts can include:
● Link to Kibana dashboards
● Aggregate counts for arbitrary fields
● Combine alerts into periodic reports
● Intercept and enhance match data
Elastalert

Elastalert

Tracing the API requests
● Trace request across services
● Down to DB statements

Going down to the DB statements
● Comments to the rescue
● SELECT "user_profiles".* FROM "user_profiles" WHERE
"user_profiles"."user_id" = '6881a8eb-5e54-4073-a2ec-a62eb4e8e746' /*
74CA2798:B364_904C6C7E:0050_5E1D733D_12189B7:0428 */
● Instrument the ORM layers to include the tracing information
● Active Record for Ruby
● adapter.prepend(::ActiveRecord::Tags::ExecuteWithTags)
● ELK stack to trace the sql requests

Why: #1 - Watchdog for the excellence
● Regularly analyze the queries, identify the weak points
Database queries taking too long based on logs-app-postgresql-2020.01.02-*
See https://kibana.showmax.cc/app/kibana#/discover/c5869b80-fe1d-11e8-a107-856e4e008c55
backend@showmax.com
download
002 1,605: DELETE FROM "download_events" WHERE "download_events"."download_id" = '<param>'
001 1,289: SELECT MAX("downloads"."updated_at") FROM "downloads" WHERE "downloads"."user_id" = '<param>'
001 1,209: SELECT "downloads".* FROM "downloads" WHERE ("downloads"."state" != '<param>') AND "downloads"."master_user_id" =
'<param>'

Why: #2 - Easy investigation of failures
● Kibana queries based on request ID are completely trackable
● Easy to analyze abnormal patterns, and track back to user
actions

Why: #3 - Auditing
● Find who messed with given data?
● Additionally to classical audit log, we are able to track the API operations
down to SQL statements.

Next steps
● Monitoring based on trends
● Create usual traffic envelope using recording rules
● Alert on anomalies in the traffic
● Loki
● “Prometheus for logs” from Grafana
● Tightly integrated with performance data
● Thanos
● HA, Long term storage, Downsampling
● Single interface to all Prometheus instances

Links
● Grafana dashboards
● https://github.com/Showmax/p2d2-2020/tree/master/dashboards
● Alert manager rules
● https://github.com/Showmax/p2d2-2020/tree/master/alerts
● PostgreSQL exporter queries.yaml
● https://github.com/Showmax/p2d2-2020/tree/master/postgres_exporter

Come and join us!
We’re looking for new colleagues
tech.showmax.com

Thanks!
Questions?
roman.fiser@showmax.com

PostgreSQL Monitoring using modern software stacks

More Related Content

What's hot (20)

Similar to PostgreSQL Monitoring using modern software stacks (20)

Recently uploaded (20)

PostgreSQL Monitoring using modern software stacks