SlideShare a Scribd company logo

Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL-Consulting LLC)

Ontico, profile picture
Ontico

The document discusses PostgreSQL statistics, emphasizing efficient usage and problem resolution based on data analysis. It covers a variety of metrics available in PostgreSQL and explores tools for gathering, interpreting, and acting on these statistics, including query performance, replication, and write activity. Additionally, it addresses the importance of understanding and leveraging statistics for database optimization.

Engineering
1 of 53
Downloaded 118 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
Deep dive into PostgreSQL Statistics. Алексей Лесовский
О чем доклад Статистика активности postgresql. Как эффективно использовать статистику. Решение проблем на основе данных статистики.
Чего НЕ будет Статистика планировщика запросов. Обзор инструментов.
Итоговая цель Умение использовать статистику это полезно. Статистика в постгресе это не страшно. Какую статистику выбрать для решения частной задачи.
Черный ящик $ ps hf -u postgres -o cmd /usr/pgsql-9.4/bin/postgres -D /var/lib/pgsql/9.4/data _ postgres: logger process _ postgres: checkpointer process _ postgres: writer process _ postgres: wal writer process _ postgres: autovacuum launcher process _ postgres: stats collector process _ postgres: postgres pgbench [local] idle in transaction _ postgres: postgres pgbench [local] idle _ postgres: postgres pgbench [local] UPDATE _ postgres: postgres pgbench [local] UPDATE waiting _ postgres: postgres pgbench [local] UPDATE
Как тратится время Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Relations IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files
Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов.
Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов. Важно уметь брать статистику напрямую. Нужны базовые знания SQL.
Что предлагают? События происходящие в базе. Свойства объектов базы (баз, таблиц, индексов). Время затраченное событием.
Источники статы Счетчики в shared memory. Функции. Встроенные представления (VIEWs). Официальный пакет contribs. Неофициальные contrib'ы.
Источники статы pg_stat_all_tables pg_current_xlog_location() pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuple pg_table_size() pg_database_size pg_index_size() pg_stat_activity EXPLAIN pg_locks pg_stat_user_functions pg_stat_all_indexes pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff() pg_stat_kcache pg_stat_statements pg_roles pg_database pg_class pg_relation_size() pg_total_relation_size() current_setting() pg_current_xlog_insert_location() pg_last_xact_replay_timestamp() pg_last_xlog_replay_location() pg_tablespace_size() pg_ls_dir()
Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements
pg_stat_database $ select * from pg_stat_database where datname = 'shop'; -[ RECORD 1 ]--+------------------------------------------- datid | 16414 datname | shop numbackends | 34 xact_commit | 51167155051 xact_rollback | 44781878826 blks_read | 7978770895 blks_hit | 9683551077519 tup_returned | 12507331807583 tup_fetched | 3885840966616 tup_inserted | 2898024857 tup_updated | 3082071349 tup_deleted | 2751363323 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 blk_read_time | 214344118.089 blk_write_time | 1260880.747 stats_reset | 2015-05-31 11:37:52.017967+03
Cache hit ratio $ select * from pg_stat_database; ... blks_read | 7978770895 blks_hit | 9683551077519 ... $ select sum(blks_hit)*100/sum(blks_hit+blks_read) as hit_ratio from pg_stat_database; Больше = лучше, но не меньше 90%
Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 xact_rollback | 44781878826 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 < 95 % xact_rollback | 44781878826 > 5 % conflicts | 0 > 10 temp_files | 377675 > 100 temp_bytes | 4783712399875 > 10 GB deadlocks | 151 > 10 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
pg_stat_bgwriter $ select * from pg_stat_bgwriter; -[ RECORD 1 ]---------+------------------------------------ checkpoints_timed | 3267 checkpoints_req | 6 checkpoint_write_time | 10416478591 checkpoint_sync_time | 405039 buffers_checkpoint | 2518065526 buffers_clean | 99602799 maxwritten_clean | 157 buffers_backend | 219356924 buffers_backend_fsync | 0 buffers_alloc | 3477374822 stats_reset | 2015-05-31 11:09:48.413185+03
Checkpoint's $ select * from pg_stat_bgwriter; ... checkpoints_timed | 3267 checkpoints_req | 6 ... checkpoints_req > checkpoints_timed = плохо postgresql.conf ● checkpoint_segments ● checkpoint_timeout ● checkpoint_completion_target
Background Writer $ select * from pg_stat_bgwriter; ... buffers_clean | 99602799 maxwritten_clean | 157 > 0 buffers_backend_fsync | 0 > 0 ... Обеспечивает бэкенды чистыми страницами. Разгружает checkpointer. postgresql.conf ● bgwriter_delay ● bgwriter_lru_maxpages ● bgwriter_lru_multiplier
pg_stat_replication $ select * from pg_stat_replication; -[ RECORD 1 ]----+------------------------------------ pid | 26921 usesysid | 15588142 usename | replica application_name | walreceiver client_addr | 10.0.0.7 client_hostname | client_port | 32956 backend_start | 2015-10-01 19:14:42.979377+03 state | streaming sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 sync_priority | 0 sync_state | async
Лаг репликации $ select * from pg_stat_replication; ... sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 ... 1692/EEE65900 — позиция в журнале транзакций. Все значения равны = идеально.
Лаг репликации Причины отставания: Networking, Storage, CPU. сколько записано xlog в байтах $ select pg_xlog_location_diff(pg_current_xlog_location(),'0/00000000'); лаг репликации в байтах $ select client_addr, pg_xlog_location_diff(pg_current_xlog_location(), replay_location) from pg_stat_replication; лаг репликации в секундах $ select extract(epoch from now() - pg_last_xact_replay_timestamp());
pg_stat_all_tables $ select * from pg_stat_all_tables; -[ RECORD 1 ]-------+------------------------------------ relid | 98221 schemaname | public relname | clientsession seq_scan | 192 seq_tup_read | 364544695 idx_scan | 2691939318 idx_tup_fetch | 2669551448 n_tup_ins | 239532851 n_tup_upd | 736119030 n_tup_del | 239898968 n_tup_hot_upd | 497688344 n_live_tup | 3493472 n_dead_tup | 1606 n_mod_since_analyze | 349526 ...
Sequential scans $ select * from pg_stat_all_tables; ... seq_scan | 192 необязательно плохо seq_tup_read | 364544695 > 1000 (seq_tup_avg) ... $ select relname, pg_size_pretty(pg_relation_size(relname::regclass)) as size, seq_scan, seq_tup_read, seq_scan / seq_tup_read as seq_tup_avg from pg_stat_user_tables where seq_tup_read > 0 order by 3,4 desc limit 5;
Размеры таблиц $ select relname, pg_size_pretty(pg_total_relation_size(relname::regclass)) as full_size, pg_size_pretty(pg_relation_size(relname::regclass)) as table_size, pg_size_pretty(pg_total_relation_size(relname::regclass) - pg_relation_size(relname::regclass)) as index_size from pg_stat_user_tables order by pg_total_relation_size(relname::regclass) desc limit 10; Также есть метакоманды psql: dt+ и di+
Write activity $ select * from pg_stat_all_tables; ... n_tup_upd | 736119030 n_tup_hot_upd | 497688344 ... UPDATE = DELETE + INSERT UPDATE = TABLE UPDATE + INDEX UPDATE
Write activity $ select s.relname, pg_size_pretty(pg_relation_size(relid)), coalesce(n_tup_ins,0) + 2 * coalesce(n_tup_upd,0) - coalesce(n_tup_hot_upd,0) + coalesce(n_tup_del,0) AS total_writes, (coalesce(n_tup_hot_upd,0)::float * 100 / (case when n_tup_upd > 0 then n_tup_upd else 1 end)::float)::numeric(10,2) AS hot_rate, (select v[1] FROM regexp_matches(reloptions::text,E'fillfactor=(d+)') as r(v) limit 1) AS fillfactor from pg_stat_all_tables s join pg_class c ON c.oid=relid order by total_writes desc limit 50; Что такое Heap-Only Tuples? HOT не вызывает перестроения индекса. HOT только для тех значений которые не участвуют в индексе Большое значение n_tup_hot_upd = хорошо. Как добиться большого n_tup_hot_upd?
Write activity | pg_size_pretty | total_writes | hot_rate | fillfactor -+----------------+--------------+----------+------------ | 9418 MB | 391459091 | 5.03 | | 417 MB | 285948760 | 0.00 | | 76 MB | 232031972 | 87.31 | 70 | 7123 MB | 124281107 | 99.36 | 70 Fillfactor определяет резерв свободного места в странице. Накладные расходы на размер таблиц (индексов). ALTER TABLE table_name SET (fillfactor = 70);
Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 85445 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100
Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 60545 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100 Avito Experience https://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue
pg_stat_all_indexes $ select * from pg_stat_all_indexes where idx_scan = 0; -[ RECORD 1 ]-+------------------------------------------ relid | 98242 indexrelid | 55732253 schemaname | public relname | products indexrelname | products_special2_idx idx_scan | 0 idx_tup_read | 0 idx_tup_fetch | 0
Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html
Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html Неиспользуемые индексы плохо. Занимают место. Замедляют UPDATE, DELETE, INSERT операции. Лишняя работа для VACUUM.
pg_stat_activity $ select * from pg_stat_activity; -[ RECORD 1 ]----+-------------------------------------------- datid | 16401 datname | ts pid | 116408 usesysid | 16384 usename | tsagent application_name | unicorn_rails worker client_addr | 172.17.0.37 client_hostname | client_port | 50888 backend_start | 2015-10-14 15:18:03.01039+00 xact_start | query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | f state | idle backend_xid | backend_xmin | query | COMMIT
Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... общий процент подключений $ select count(*)*100/(select current_setting('max_connections')::int) from pg_stat_activity;
Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... $ select client_addr, usename, datname, count(*) from pg_stat_activity group by 1,2,3 order by 4 desc; client_addr | usename | datname | count -------------+-----------+---------+------- 127.0.0.1 | cron_role | ts | 508 127.0.0.1 | tsagent | ts | 60 127.0.0.1 | postgres | ts | 1 127.0.0.1 | pupkin | ts | 1
Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ...
Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start;
Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start; clock_timestamp() для вычисления времени работы. Запросы запоминаем, отстреливаем, оптимизируем (EXPLAIN).
Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ...
Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ... idle in transaction, idle in transaction (aborted) = плохо Повод для беспокойств: > 5 clock_timestamp() для вычисления времени работы. Транзакции отстреливаем, приложение оптимизируем.
Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ...
Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ... waiting = true = плохо. clock_timestamp() для вычисления времени работы. pg_locks для поиска источника блокировки. Транзакции отстреливаем, приложение оптимизируем.
Блокировки https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_sho w_locked_queries.sql http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/ locked_item | products waiting | t waiting_query | update products set avail = false where id = 154; waiting_mode | RowExclusiveLock waiting_xact_age | 00:00:07.103685 waiting_query_age | 00:00:01.813508 waiting_pid | 5895 waiting_user | app_role waiting_state | active waiting | f locking_query | alter table products drop column avail; locking_mode | AccessExclusiveLock locking_xact_age | 00:00:16.416441 locking_query_age | 00:00:14.416441 locking_pid | 5844 locking_user | postgres locking_state | idle in transaction
pg_stat_statements $ select * from pg_stat_statements where blk_read_time <> 0 order by blk_read_time desc; -[ RECORD 1 ]-------+--------------------------------------- userid | 25078444 dbid | 16411 query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 shared_blks_hit | 351353214 shared_blks_read | 205557 shared_blks_dirtied | 256053 shared_blks_written | 0 local_blks_hit | 0 local_blks_read | 68894 local_blks_dirtied | 68894 local_blks_written | 0 temp_blks_read | 0 temp_blks_written | 0 blk_read_time | 495425.535999976 blk_write_time | 0
pg_stat_statements $ select * from pg_stat_statements; ... query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 blk_read_time | 495425.535999976 blk_write_time | 0 Cреднее время запроса в милисекундах $ select (sum(total_time) / sum(calls))::numeric(6,3) from pg_stat_statements; Самые активно пишущие (в shared_buffers) запросы $ select query, shared_blks_dirtied from pg_stat_statements where shared_blks_dirtied > 0 order by 2 desc;
Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/glob al_reports/query_stat_total.sql
Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 Используем sum() для подсчет общей статистики. Вычисляем «вклад» запроса в общую статистику. Использование ресурсов (CPU, IO).
За кадром pg_statio_all_tables, pg_statio_all_indexes pg_stat_user_functions Функции определения размеров - df *size* pgstattuple (contrib) ● точное определение bloat для индексов и таблиц ● чем больше размер таблицы, тем больше времени уйдет на оценку pg_buffercache (contrib) ● инспекция shared buffers ● большие накладые расходы (buffers lock)
За кадром pgfincore ● низкоуровневые операции с таблицами через mincore(). ● инспекция OS page cache. pg_stat_kcache ● использование getrusage() до и после запроса. ● cpu usage и дисковый ввод-вывод. ● требует pg_stat_statements и postgresql-9.4. ● почти не влияет на производительность.
Резюме ● Умение пользоваться статистикой полезно. ● Статистика в постгресе это не сложно. ● Статистика поможет ответить на вопросы. ● Экспериментируйте.
Годные ссылки Автор пиши ещё www.blog.postgresql-consulting.com (eng) www.thislinux.org (rus) The Statistics Collector http://www.postgresql.org/docs/current/static/monitoring-stats.html System Administration Functions http://www.postgresql.org/docs/current/static/functions-admin.html Contrib modules http://www.postgresql.org/docs/current/static/pgstatstatements.html http://www.postgresql.org/docs/current/static/pgstattuple.html http://www.postgresql.org/docs/current/static/pgbuffercache.html https://github.com/klando/pgfincore https://github.com/dalibo/pg_stat_kcache SQL utils and sql code examples https://github.com/PostgreSQL-Consulting/pg-utils
Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements Спасибо. Вопросы?

More Related Content

PDF
Troubleshooting PostgreSQL with pgCenter
Alexey Lesovsky
 
PDF
Managing PostgreSQL with PgCenter
Alexey Lesovsky
 
PDF
PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).
Alexey Lesovsky
 
PDF
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)
Ontico
 
PDF
Advanced Postgres Monitoring
Denish Patel
 
PDF
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
PPTX
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
Jeff Frost
 
PDF
GitLab PostgresMortem: Lessons Learned
Alexey Lesovsky
 
Troubleshooting PostgreSQL with pgCenter
Alexey Lesovsky
 
Managing PostgreSQL with PgCenter
Alexey Lesovsky
 
PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).
Alexey Lesovsky
 
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)
Ontico
 
Advanced Postgres Monitoring
Denish Patel
 
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
Jeff Frost
 
GitLab PostgresMortem: Lessons Learned
Alexey Lesovsky
 

What's hot (20)

PDF
Strategic autovacuum
Jim Mlodgenski
 
ODP
PostgreSQL Administration for System Administrators
Command Prompt., Inc
 
PDF
PostgreSQL query planner's internals
Alexey Ermakov
 
PDF
Peeking into the Black Hole Called PL/PGSQL - the New PL Profiler / Jan Wieck...
Ontico
 
PDF
PostgreSQL Streaming Replication Cheatsheet
Alexey Lesovsky
 
PDF
Как PostgreSQL работает с диском
PostgreSQL-Consulting
 
PDF
PostgreSQL Replication Tutorial
Hans-Jürgen Schönig
 
PDF
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Jim Mlodgenski
 
PDF
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...
Ontico
 
PDF
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Badoo Development
 
PDF
Out of the box replication in postgres 9.4
Denish Patel
 
PDF
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Ontico
 
PDF
WiredTiger In-Memory vs WiredTiger B-Tree
Sveta Smirnova
 
PDF
Open Source SQL databases enters millions queries per second era
Sveta Smirnova
 
PDF
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Ontico
 
PDF
XtraDB 5.7: key performance algorithms
Laurynas Biveinis
 
PDF
pg / shardman: шардинг в PostgreSQL на основе postgres / fdw, pg / pathman и ...
Ontico
 
PDF
Sessionization with Spark streaming
Ramūnas Urbonas
 
PDF
Linux Systems Performance 2016
Brendan Gregg
 
PDF
Performance Profiling in Rust
InfluxData
 
Strategic autovacuum
Jim Mlodgenski
 
PostgreSQL Administration for System Administrators
Command Prompt., Inc
 
PostgreSQL query planner's internals
Alexey Ermakov
 
Peeking into the Black Hole Called PL/PGSQL - the New PL Profiler / Jan Wieck...
Ontico
 
PostgreSQL Streaming Replication Cheatsheet
Alexey Lesovsky
 
Как PostgreSQL работает с диском
PostgreSQL-Consulting
 
PostgreSQL Replication Tutorial
Hans-Jürgen Schönig
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Jim Mlodgenski
 
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...
Ontico
 
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Badoo Development
 
Out of the box replication in postgres 9.4
Denish Patel
 
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Ontico
 
WiredTiger In-Memory vs WiredTiger B-Tree
Sveta Smirnova
 
Open Source SQL databases enters millions queries per second era
Sveta Smirnova
 
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Ontico
 
XtraDB 5.7: key performance algorithms
Laurynas Biveinis
 
pg / shardman: шардинг в PostgreSQL на основе postgres / fdw, pg / pathman и ...
Ontico
 
Sessionization with Spark streaming
Ramūnas Urbonas
 
Linux Systems Performance 2016
Brendan Gregg
 
Performance Profiling in Rust
InfluxData
 
Ad

Viewers also liked (18)

PPTX
Мониторинг всех слоев web проекта / Сивко Николай (hh.ru)
Ontico
 
PPTX
Chronicle Map — key-value хранилище для трейдинга на Java / Левентов Роман (C...
Ontico
 
PDF
Где живут Ваши объявления / Тюрин Михаил (Avito)
Ontico
 
PDF
Мониторинг ожиданий в PostgreSQL / Курбангалиев Ильдус (Postgres Professional)
Ontico
 
PDF
PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
Ontico
 
PPTX
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)
Ontico
 
PDF
Что нового и полезного в PostgreSQL 9.5 / Илья Космодемьянский (PostgreSQL-Co...
Ontico
 
PDF
PostgreSQL в высоконагруженных проектах
Alexey Vasiliev
 
PPTX
Учебный план для highload гуру / Андрей Аксёнов (Sphinx Technologies Inc.)
Ontico
 
PDF
Сага о кластере. Все что вы хотели знать про горизонтальное масштабирование в...
Ontico
 
PPTX
Настройка и оптимизация высоконагруженных J2EE веб-приложений / Шамим Ахмед (...
Ontico
 
PDF
Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)
Ontico
 
PDF
NodeJS в HighLoad проекте / Акрицкий Владимир (iAge Engineering)
Ontico
 
PDF
Docker networking
Badoo Development
 
PDF
Мониторь, автоматизируй Docker
Badoo Development
 
PDF
Docker в Badoo: ПМЖ или временная регистрация
Badoo Development
 
PDF
Docker penetration
Badoo Development
 
PDF
10 SQL Tricks that You Didn't Think Were Possible
Lukas Eder
 
Мониторинг всех слоев web проекта / Сивко Николай (hh.ru)
Ontico
 
Chronicle Map — key-value хранилище для трейдинга на Java / Левентов Роман (C...
Ontico
 
Где живут Ваши объявления / Тюрин Михаил (Avito)
Ontico
 
Мониторинг ожиданий в PostgreSQL / Курбангалиев Ильдус (Postgres Professional)
Ontico
 
PostgreSQL - масштабирование в моде, Valentine Gogichashvili (Zalando SE)
Ontico
 
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)
Ontico
 
Что нового и полезного в PostgreSQL 9.5 / Илья Космодемьянский (PostgreSQL-Co...
Ontico
 
PostgreSQL в высоконагруженных проектах
Alexey Vasiliev
 
Учебный план для highload гуру / Андрей Аксёнов (Sphinx Technologies Inc.)
Ontico
 
Сага о кластере. Все что вы хотели знать про горизонтальное масштабирование в...
Ontico
 
Настройка и оптимизация высоконагруженных J2EE веб-приложений / Шамим Ахмед (...
Ontico
 
Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)
Ontico
 
NodeJS в HighLoad проекте / Акрицкий Владимир (iAge Engineering)
Ontico
 
Docker networking
Badoo Development
 
Мониторь, автоматизируй Docker
Badoo Development
 
Docker в Badoo: ПМЖ или временная регистрация
Badoo Development
 
Docker penetration
Badoo Development
 
10 SQL Tricks that You Didn't Think Were Possible
Lukas Eder
 
Ad

Similar to Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL-Consulting LLC) (20)

PDF
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
PDF
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
PPTX
Postgresql Database Administration- Day4
PoguttuezhiniVP
 
PDF
Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl
Citus Data
 
PDF
Explain this!
Fabio Telles Rodriguez
 
PDF
PGConf APAC 2018 - Monitoring PostgreSQL at Scale
PGConf APAC
 
PPTX
Elegant database tuning
Scott Mead
 
PDF
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Citus Data
 
PDF
Optimizing your app by understanding your Postgres | RailsConf 2019 | Samay S...
Citus Data
 
PDF
Aplicações 10x a 100x mais rápida com o postgre sql
Fabio Telles Rodriguez
 
PDF
2015.07.16 Способы диагностики PostgreSQL
dev1ant
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PDF
Using PostgreSQL statistics to optimize performance
Alexey Ermakov
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
PDF
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
PDF
PostgreSQL on Solaris
Theo Schlossnagle
 
PDF
PostgreSQL on Solaris
Theo Schlossnagle
 
PPTX
Postgresql Database Administration Basic - Day2
PoguttuezhiniVP
 
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Postgresql Database Administration- Day4
PoguttuezhiniVP
 
Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl
Citus Data
 
Explain this!
Fabio Telles Rodriguez
 
PGConf APAC 2018 - Monitoring PostgreSQL at Scale
PGConf APAC
 
Elegant database tuning
Scott Mead
 
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Citus Data
 
Optimizing your app by understanding your Postgres | RailsConf 2019 | Samay S...
Citus Data
 
Aplicações 10x a 100x mais rápida com o postgre sql
Fabio Telles Rodriguez
 
2015.07.16 Способы диагностики PostgreSQL
dev1ant
 
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
Using PostgreSQL statistics to optimize performance
Alexey Ermakov
 
pg_proctab: Accessing System Stats in PostgreSQL
Mark Wong
 
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
PostgreSQL on Solaris
Theo Schlossnagle
 
PostgreSQL on Solaris
Theo Schlossnagle
 
Postgresql Database Administration Basic - Day2
PoguttuezhiniVP
 

More from Ontico (20)

PDF
One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...
Ontico
 
PDF
Масштабируя DNS / Артем Гавриченков (Qrator Labs)
Ontico
 
PPTX
Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)
Ontico
 
PDF
Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...
Ontico
 
PDF
Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...
Ontico
 
PDF
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
Ontico
 
PDF
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Ontico
 
PDF
Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...
Ontico
 
PPTX
ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)
Ontico
 
PPTX
MySQL Replication — Advanced Features / Петр Зайцев (Percona)
Ontico
 
PDF
Внутренний open-source. Как разрабатывать мобильное приложение большим количе...
Ontico
 
PPTX
Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...
Ontico
 
PPTX
Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...
Ontico
 
PDF
Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)
Ontico
 
PPT
И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)
Ontico
 
PPTX
Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)
Ontico
 
PPTX
Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)
Ontico
 
PPTX
100500 способов кэширования в Oracle Database или как достичь максимальной ск...
Ontico
 
PPTX
Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...
Ontico
 
PDF
Механизмы мониторинга баз данных: взгляд изнутри / Дмитрий Еманов (Firebird P...
Ontico
 
One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...
Ontico
 
Масштабируя DNS / Артем Гавриченков (Qrator Labs)
Ontico
 
Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)
Ontico
 
Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...
Ontico
 
Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...
Ontico
 
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
Ontico
 
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Ontico
 
Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...
Ontico
 
ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)
Ontico
 
MySQL Replication — Advanced Features / Петр Зайцев (Percona)
Ontico
 
Внутренний open-source. Как разрабатывать мобильное приложение большим количе...
Ontico
 
Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...
Ontico
 
Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...
Ontico
 
Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)
Ontico
 
И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)
Ontico
 
Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)
Ontico
 
Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)
Ontico
 
100500 способов кэширования в Oracle Database или как достичь максимальной ск...
Ontico
 
Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...
Ontico
 
Механизмы мониторинга баз данных: взгляд изнутри / Дмитрий Еманов (Firebird P...
Ontico
 

Recently uploaded (20)

PPTX
Construction Project Organization Group 2.pptx
vj5agdales
 
PPTX
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
PPTX
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
PPTX
Strings in CPP - Strings in C++ are sequences of characters used to store and...
sangeethamtech26
 
PDF
BMEC211 - INTRODUCTION TO MECHATRONICS-1.pdf
ryankakungu
 
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
PPTX
web development for engineering and engineering
keshriji700
 
PPTX
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
PDF
Arduino robotics embedded978-1-4302-3184-4.pdf
AbdullahEmam4
 
PPTX
Welding lecture in detail for understanding
sahilpoonia10
 
PDF
Structs to JSON How Go Powers REST APIs.pdf
Emily Achieng
 
PPTX
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
PPTX
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
abhi1361yadav
 
PPT
Project quality management in manufacturing
BarMusaTetik
 
PDF
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
PDF
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
IJAEMSJORNAL
 
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
DOCX
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
AYUSHMANAV
 
PDF
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
PDF
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
sebastianku31
 
Construction Project Organization Group 2.pptx
vj5agdales
 
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
Strings in CPP - Strings in C++ are sequences of characters used to store and...
sangeethamtech26
 
BMEC211 - INTRODUCTION TO MECHATRONICS-1.pdf
ryankakungu
 
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
web development for engineering and engineering
keshriji700
 
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
Arduino robotics embedded978-1-4302-3184-4.pdf
AbdullahEmam4
 
Welding lecture in detail for understanding
sahilpoonia10
 
Structs to JSON How Go Powers REST APIs.pdf
Emily Achieng
 
OOP with Java - Java Introduction (Basics)
Rashmi T V
 
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
abhi1361yadav
 
Project quality management in manufacturing
BarMusaTetik
 
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
IJAEMSJORNAL
 
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
ASol_English-Language-Literature-Set-1-27-02-2023-converted.docx
AYUSHMANAV
 
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
sebastianku31
 

Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL-Consulting LLC)

  • 2. О чем доклад Статистика активности postgresql. Как эффективно использовать статистику. Решение проблем на основе данных статистики.
  • 3. Чего НЕ будет Статистика планировщика запросов. Обзор инструментов.
  • 4. Итоговая цель Умение использовать статистику это полезно. Статистика в постгресе это не страшно. Какую статистику выбрать для решения частной задачи.
  • 5. Черный ящик $ ps hf -u postgres -o cmd /usr/pgsql-9.4/bin/postgres -D /var/lib/pgsql/9.4/data _ postgres: logger process _ postgres: checkpointer process _ postgres: writer process _ postgres: wal writer process _ postgres: autovacuum launcher process _ postgres: stats collector process _ postgres: postgres pgbench [local] idle in transaction _ postgres: postgres pgbench [local] idle _ postgres: postgres pgbench [local] UPDATE _ postgres: postgres pgbench [local] UPDATE waiting _ postgres: postgres pgbench [local] UPDATE
  • 6. Как тратится время Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Relations IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files
  • 7. Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов.
  • 8. Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов. Важно уметь брать статистику напрямую. Нужны базовые знания SQL.
  • 9. Что предлагают? События происходящие в базе. Свойства объектов базы (баз, таблиц, индексов). Время затраченное событием.
  • 10. Источники статы Счетчики в shared memory. Функции. Встроенные представления (VIEWs). Официальный пакет contribs. Неофициальные contrib'ы.
  • 12. Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements
  • 13. pg_stat_database $ select * from pg_stat_database where datname = 'shop'; -[ RECORD 1 ]--+------------------------------------------- datid | 16414 datname | shop numbackends | 34 xact_commit | 51167155051 xact_rollback | 44781878826 blks_read | 7978770895 blks_hit | 9683551077519 tup_returned | 12507331807583 tup_fetched | 3885840966616 tup_inserted | 2898024857 tup_updated | 3082071349 tup_deleted | 2751363323 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 blk_read_time | 214344118.089 blk_write_time | 1260880.747 stats_reset | 2015-05-31 11:37:52.017967+03
  • 14. Cache hit ratio $ select * from pg_stat_database; ... blks_read | 7978770895 blks_hit | 9683551077519 ... $ select sum(blks_hit)*100/sum(blks_hit+blks_read) as hit_ratio from pg_stat_database; Больше = лучше, но не меньше 90%
  • 15. Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 xact_rollback | 44781878826 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
  • 16. Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 < 95 % xact_rollback | 44781878826 > 5 % conflicts | 0 > 10 temp_files | 377675 > 100 temp_bytes | 4783712399875 > 10 GB deadlocks | 151 > 10 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
  • 17. pg_stat_bgwriter $ select * from pg_stat_bgwriter; -[ RECORD 1 ]---------+------------------------------------ checkpoints_timed | 3267 checkpoints_req | 6 checkpoint_write_time | 10416478591 checkpoint_sync_time | 405039 buffers_checkpoint | 2518065526 buffers_clean | 99602799 maxwritten_clean | 157 buffers_backend | 219356924 buffers_backend_fsync | 0 buffers_alloc | 3477374822 stats_reset | 2015-05-31 11:09:48.413185+03
  • 18. Checkpoint's $ select * from pg_stat_bgwriter; ... checkpoints_timed | 3267 checkpoints_req | 6 ... checkpoints_req > checkpoints_timed = плохо postgresql.conf ● checkpoint_segments ● checkpoint_timeout ● checkpoint_completion_target
  • 19. Background Writer $ select * from pg_stat_bgwriter; ... buffers_clean | 99602799 maxwritten_clean | 157 > 0 buffers_backend_fsync | 0 > 0 ... Обеспечивает бэкенды чистыми страницами. Разгружает checkpointer. postgresql.conf ● bgwriter_delay ● bgwriter_lru_maxpages ● bgwriter_lru_multiplier
  • 20. pg_stat_replication $ select * from pg_stat_replication; -[ RECORD 1 ]----+------------------------------------ pid | 26921 usesysid | 15588142 usename | replica application_name | walreceiver client_addr | 10.0.0.7 client_hostname | client_port | 32956 backend_start | 2015-10-01 19:14:42.979377+03 state | streaming sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 sync_priority | 0 sync_state | async
  • 21. Лаг репликации $ select * from pg_stat_replication; ... sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 ... 1692/EEE65900 — позиция в журнале транзакций. Все значения равны = идеально.
  • 22. Лаг репликации Причины отставания: Networking, Storage, CPU. сколько записано xlog в байтах $ select pg_xlog_location_diff(pg_current_xlog_location(),'0/00000000'); лаг репликации в байтах $ select client_addr, pg_xlog_location_diff(pg_current_xlog_location(), replay_location) from pg_stat_replication; лаг репликации в секундах $ select extract(epoch from now() - pg_last_xact_replay_timestamp());
  • 23. pg_stat_all_tables $ select * from pg_stat_all_tables; -[ RECORD 1 ]-------+------------------------------------ relid | 98221 schemaname | public relname | clientsession seq_scan | 192 seq_tup_read | 364544695 idx_scan | 2691939318 idx_tup_fetch | 2669551448 n_tup_ins | 239532851 n_tup_upd | 736119030 n_tup_del | 239898968 n_tup_hot_upd | 497688344 n_live_tup | 3493472 n_dead_tup | 1606 n_mod_since_analyze | 349526 ...
  • 24. Sequential scans $ select * from pg_stat_all_tables; ... seq_scan | 192 необязательно плохо seq_tup_read | 364544695 > 1000 (seq_tup_avg) ... $ select relname, pg_size_pretty(pg_relation_size(relname::regclass)) as size, seq_scan, seq_tup_read, seq_scan / seq_tup_read as seq_tup_avg from pg_stat_user_tables where seq_tup_read > 0 order by 3,4 desc limit 5;
  • 25. Размеры таблиц $ select relname, pg_size_pretty(pg_total_relation_size(relname::regclass)) as full_size, pg_size_pretty(pg_relation_size(relname::regclass)) as table_size, pg_size_pretty(pg_total_relation_size(relname::regclass) - pg_relation_size(relname::regclass)) as index_size from pg_stat_user_tables order by pg_total_relation_size(relname::regclass) desc limit 10; Также есть метакоманды psql: dt+ и di+
  • 26. Write activity $ select * from pg_stat_all_tables; ... n_tup_upd | 736119030 n_tup_hot_upd | 497688344 ... UPDATE = DELETE + INSERT UPDATE = TABLE UPDATE + INDEX UPDATE
  • 27. Write activity $ select s.relname, pg_size_pretty(pg_relation_size(relid)), coalesce(n_tup_ins,0) + 2 * coalesce(n_tup_upd,0) - coalesce(n_tup_hot_upd,0) + coalesce(n_tup_del,0) AS total_writes, (coalesce(n_tup_hot_upd,0)::float * 100 / (case when n_tup_upd > 0 then n_tup_upd else 1 end)::float)::numeric(10,2) AS hot_rate, (select v[1] FROM regexp_matches(reloptions::text,E'fillfactor=(d+)') as r(v) limit 1) AS fillfactor from pg_stat_all_tables s join pg_class c ON c.oid=relid order by total_writes desc limit 50; Что такое Heap-Only Tuples? HOT не вызывает перестроения индекса. HOT только для тех значений которые не участвуют в индексе Большое значение n_tup_hot_upd = хорошо. Как добиться большого n_tup_hot_upd?
  • 28. Write activity | pg_size_pretty | total_writes | hot_rate | fillfactor -+----------------+--------------+----------+------------ | 9418 MB | 391459091 | 5.03 | | 417 MB | 285948760 | 0.00 | | 76 MB | 232031972 | 87.31 | 70 | 7123 MB | 124281107 | 99.36 | 70 Fillfactor определяет резерв свободного места в странице. Накладные расходы на размер таблиц (индексов). ALTER TABLE table_name SET (fillfactor = 70);
  • 29. Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 85445 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100
  • 30. Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 60545 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100 Avito Experience https://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue
  • 31. pg_stat_all_indexes $ select * from pg_stat_all_indexes where idx_scan = 0; -[ RECORD 1 ]-+------------------------------------------ relid | 98242 indexrelid | 55732253 schemaname | public relname | products indexrelname | products_special2_idx idx_scan | 0 idx_tup_read | 0 idx_tup_fetch | 0
  • 32. Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html
  • 33. Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html Неиспользуемые индексы плохо. Занимают место. Замедляют UPDATE, DELETE, INSERT операции. Лишняя работа для VACUUM.
  • 34. pg_stat_activity $ select * from pg_stat_activity; -[ RECORD 1 ]----+-------------------------------------------- datid | 16401 datname | ts pid | 116408 usesysid | 16384 usename | tsagent application_name | unicorn_rails worker client_addr | 172.17.0.37 client_hostname | client_port | 50888 backend_start | 2015-10-14 15:18:03.01039+00 xact_start | query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | f state | idle backend_xid | backend_xmin | query | COMMIT
  • 35. Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... общий процент подключений $ select count(*)*100/(select current_setting('max_connections')::int) from pg_stat_activity;
  • 36. Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... $ select client_addr, usename, datname, count(*) from pg_stat_activity group by 1,2,3 order by 4 desc; client_addr | usename | datname | count -------------+-----------+---------+------- 127.0.0.1 | cron_role | ts | 508 127.0.0.1 | tsagent | ts | 60 127.0.0.1 | postgres | ts | 1 127.0.0.1 | pupkin | ts | 1
  • 37. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ...
  • 38. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start;
  • 39. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start; clock_timestamp() для вычисления времени работы. Запросы запоминаем, отстреливаем, оптимизируем (EXPLAIN).
  • 40. Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ...
  • 41. Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ... idle in transaction, idle in transaction (aborted) = плохо Повод для беспокойств: > 5 clock_timestamp() для вычисления времени работы. Транзакции отстреливаем, приложение оптимизируем.
  • 42. Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ...
  • 43. Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ... waiting = true = плохо. clock_timestamp() для вычисления времени работы. pg_locks для поиска источника блокировки. Транзакции отстреливаем, приложение оптимизируем.
  • 44. Блокировки https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_sho w_locked_queries.sql http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/ locked_item | products waiting | t waiting_query | update products set avail = false where id = 154; waiting_mode | RowExclusiveLock waiting_xact_age | 00:00:07.103685 waiting_query_age | 00:00:01.813508 waiting_pid | 5895 waiting_user | app_role waiting_state | active waiting | f locking_query | alter table products drop column avail; locking_mode | AccessExclusiveLock locking_xact_age | 00:00:16.416441 locking_query_age | 00:00:14.416441 locking_pid | 5844 locking_user | postgres locking_state | idle in transaction
  • 45. pg_stat_statements $ select * from pg_stat_statements where blk_read_time <> 0 order by blk_read_time desc; -[ RECORD 1 ]-------+--------------------------------------- userid | 25078444 dbid | 16411 query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 shared_blks_hit | 351353214 shared_blks_read | 205557 shared_blks_dirtied | 256053 shared_blks_written | 0 local_blks_hit | 0 local_blks_read | 68894 local_blks_dirtied | 68894 local_blks_written | 0 temp_blks_read | 0 temp_blks_written | 0 blk_read_time | 495425.535999976 blk_write_time | 0
  • 46. pg_stat_statements $ select * from pg_stat_statements; ... query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 blk_read_time | 495425.535999976 blk_write_time | 0 Cреднее время запроса в милисекундах $ select (sum(total_time) / sum(calls))::numeric(6,3) from pg_stat_statements; Самые активно пишущие (в shared_buffers) запросы $ select query, shared_blks_dirtied from pg_stat_statements where shared_blks_dirtied > 0 order by 2 desc;
  • 47. Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/glob al_reports/query_stat_total.sql
  • 48. Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 Используем sum() для подсчет общей статистики. Вычисляем «вклад» запроса в общую статистику. Использование ресурсов (CPU, IO).
  • 49. За кадром pg_statio_all_tables, pg_statio_all_indexes pg_stat_user_functions Функции определения размеров - df *size* pgstattuple (contrib) ● точное определение bloat для индексов и таблиц ● чем больше размер таблицы, тем больше времени уйдет на оценку pg_buffercache (contrib) ● инспекция shared buffers ● большие накладые расходы (buffers lock)
  • 50. За кадром pgfincore ● низкоуровневые операции с таблицами через mincore(). ● инспекция OS page cache. pg_stat_kcache ● использование getrusage() до и после запроса. ● cpu usage и дисковый ввод-вывод. ● требует pg_stat_statements и postgresql-9.4. ● почти не влияет на производительность.
  • 51. Резюме ● Умение пользоваться статистикой полезно. ● Статистика в постгресе это не сложно. ● Статистика поможет ответить на вопросы. ● Экспериментируйте.
  • 52. Годные ссылки Автор пиши ещё www.blog.postgresql-consulting.com (eng) www.thislinux.org (rus) The Statistics Collector http://www.postgresql.org/docs/current/static/monitoring-stats.html System Administration Functions http://www.postgresql.org/docs/current/static/functions-admin.html Contrib modules http://www.postgresql.org/docs/current/static/pgstatstatements.html http://www.postgresql.org/docs/current/static/pgstattuple.html http://www.postgresql.org/docs/current/static/pgbuffercache.html https://github.com/klando/pgfincore https://github.com/dalibo/pg_stat_kcache SQL utils and sql code examples https://github.com/PostgreSQL-Consulting/pg-utils
  • 53. Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements Спасибо. Вопросы?