MySQL Monitoring using Prometheus & Grafana
Download as PPTX, PDF11 likes4,515 views
This document describes how to set up monitoring for MySQL databases using Prometheus and Grafana. It includes instructions for installing and configuring Prometheus and Alertmanager on a monitoring server to scrape metrics from node_exporter and mysql_exporter. Ansible playbooks are provided to automatically install the exporters and configure Prometheus. Finally, steps are outlined for creating Grafana dashboards to visualize the metrics and monitor MySQL performance.
![Install & config prometheus on monitoring server
6
2. Config prometheus.yml file
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: linux_hostname1
target_groups:
- targets: [‘ip_addr:9100']
labels:
alias: hostname1
- job_name: mysql_hostname1
target_groups:
- targets: [‘ip_addr:9104']
labels:
alias: hostname1
/opt/prometheus/prometheus.yml
continue right side…
scrape_configs :
you can add server information.
It can be added automatically using ansible playbook.](https://image.slidesharecdn.com/mysqlmonitoring20161012-161020075611/85/MySQL-Monitoring-using-Prometheus-Grafana-6-320.jpg)
![Install & config alertmanager on monitoring server
8
3. Config simple.yml
/opt/alertmanager/simple.yml
global:
slack_api_url: {{ webhook url}}
route:
receiver: 'slack-notifications'
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 1m
repeat_interval: 3m
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: {{ slack channel name}}
continue right side…](https://image.slidesharecdn.com/mysqlmonitoring20161012-161020075611/85/MySQL-Monitoring-using-Prometheus-Grafana-8-320.jpg)
![19
/etc/hosts
~/ansible_hosts
#DB server lists
10.xxx.xxx.xxx dbhost01
10.xxx.xxx.xxx dbhost02
10.xxx.xxx.xxx dbhost03
[Group1]
dbhost01
dbhost02
dbhost03
system hosts file
ansible hosts file
You should set following information on profile.
export ANSIBLE_INVENTORY=~/ansible_hosts
Auto install & config exporters using Ansible](https://image.slidesharecdn.com/mysqlmonitoring20161012-161020075611/85/MySQL-Monitoring-using-Prometheus-Grafana-19-320.jpg)
![25
~/ansible/playbooks/files/prometheus/
.my.cnf
[client]
host= {{ansible_default_ipv4.address}}
user=mon_user # db user for monitoring
password=mon_user_passwd # db user password
Auto install & config exporters using Ansible
: mysqld_exporter config file
mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO ‘mon_user'@'127.0.0.1' identified by ‘mon_user_passwd';
mysql> GRANT SELECT ON performance_schema.* TO ‘mon_user'@'127.0.0.1';
You have to add db user for monitoring on db server](https://image.slidesharecdn.com/mysqlmonitoring20161012-161020075611/85/MySQL-Monitoring-using-Prometheus-Grafana-25-320.jpg)
![26
~/ansible/playbooks/files/prometheus/
prometheus.yml.temp
- job_name: linux_{{ansible_hostname}}
target_groups:
- targets: ['{{ansible_default_ipv4.address}}:9100']
labels:
alias: {{ansible_hostname}}
- job_name: mysql_{{ansible_hostname}}
target_groups:
- targets: ['{{ansible_default_ipv4.address}}:9104']
labels:
alias: {{ansible_hostname}}
Auto install & config exporters using Ansible
: prometheus config information](https://image.slidesharecdn.com/mysqlmonitoring20161012-161020075611/85/MySQL-Monitoring-using-Prometheus-Grafana-26-320.jpg)
MySQL Monitoring using Prometheus & Grafana
- 1. MySQL Monitoring using Prometheus & Grafana
- 2. 2 dba.kim@gmail.com https://kr.linkedin.com/in/youngheon-roy-kim-23a11181 About me
- 3. MySQL Monitoring Dashboard 3 Alert using slack Grafana Dashboard
- 4. Architecture 4 DB Servers Monitoring Server node_exporter mysql_exporter prometheus alert manager playbook node_exporter mysql_exporter • • • Deploy Server
- 5. Install & config prometheus on monitoring server 5 1. Install Prometheus Download .tar file https://prometheus.io/download/ $tar –xvf prometheus-0.17.0rc2.linux-amd64.tar.gz –C /opt $mv prometheus-0.17.0rc2.linux-amd64 prometheus
- 6. Install & config prometheus on monitoring server 6 2. Config prometheus.yml file global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: linux_hostname1 target_groups: - targets: [‘ip_addr:9100'] labels: alias: hostname1 - job_name: mysql_hostname1 target_groups: - targets: [‘ip_addr:9104'] labels: alias: hostname1 /opt/prometheus/prometheus.yml continue right side… scrape_configs : you can add server information. It can be added automatically using ansible playbook.
- 7. Install & config alertmanager on monitoring server 7 1. Install alertmanager Download .tar file https://prometheus.io/download/ $tar –xvf alertmanager-0.1.1.linux-amd64.tar.gz –C /opt $mv alertmanager-0.1.1.linux-amd64 alertmanager 2. make a slack Webhook url You can make a slack webhook url for sending a alert message. https://slack.com/apps/A0F7XDUAZ-incoming-webhooks
- 8. Install & config alertmanager on monitoring server 8 3. Config simple.yml /opt/alertmanager/simple.yml global: slack_api_url: {{ webhook url}} route: receiver: 'slack-notifications' group_by: ['alertname', 'cluster', 'service'] group_wait: 30s group_interval: 1m repeat_interval: 3m receivers: - name: 'slack-notifications' slack_configs: - channel: {{ slack channel name}} continue right side…
- 9. 9 4. Config rule file(1/4) /opt/alertmanager/alert.rules ALERT LinuxLoad1m IF node_load1 >15 FOR 1m ANNOTATIONS {summary="Linux Load is at 10"} ALERT LinuxMemory IF ((node_memory_MemTotal - (node_memory_MemFree + node_memory_Buffers + node_memory_Cached))/node_memory_MemTotal) * 100 >85 FOR 1m ANNOTATIONS {summary="Linux Memory Usage is at 85%"} continue next page… Set a alert rule file Install & config alertmanager on monitoring server
- 10. 10 4. Config rule file(2/4) /opt/alertmanager/alert.rules ALERT LinuxDiskUsage IF ((node_filesystem_size{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"} - node_filesystem_avail{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"}) /node_filesystem_size{ fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs"}) * 100 > 90 FOR 5m ANNOTATIONS {summary="Linux Disk Usage over 90%"} ALERT MySQLSlaveLag IF mysql_slave_lag_seconds > 300 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too high.", severity="warning" } Set a alert rule file Install & config alertmanager on monitoring server continue next page…
- 11. 11 4. Config rule file(3/4) /opt/alertmanager/alert.rules ALERT MySQLReplicationSQLThreadStatus IF mysql_slave_status_slave_sql_running==0 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "SQL thread stop", severity="warning"} ALERT MySQLReplicationIOThreadStatus IF mysql_slave_status_slave_io_running==0 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "IO thread stop", severity="warning"} continue next page… Set a alert rule file Install & config alertmanager on monitoring server
- 12. 12 4. Config rule file(4/4) /opt/alertmanager/alert.rules ALERT MySQLstatus IF mysql_up==0 FOR 30s LABELS { severity = "warning" } ANNOTATIONS { summary = "Mysql Process Down" } continue next page… Set a alert rule file Install & config alertmanager on monitoring server
- 13. 13 5. Add rule file information on prometheus.yml /opt/prometheus/prometheus.yml You can add following information between global and scrape_configs section. global: rule_files: - /opt/alertmanager/alert.rules scrape_configs: Set a alert rule file Install & config alertmanager on monitoring server
- 14. 14 6. Start alertmanager and prometheus Alertmanager start #./alertmanager –config.file=simple.yml Prometheus start #./prometheus –config.file prometheus.yml –alertmanager.url=http://10.xxx.xxx.xxx:9093 start alertmanager and prometheus Install & config alertmanager on monitoring server
- 15. Install & config alertmanager on monitoring server 15 7. Check status of alertmanager and prometheus Prometheus status http://10.xxx.xxx.xxx:9090/status
- 16. Install & config alertmanager on monitoring server 16 7. Check status of alertmanager and prometheus Alertmanager status http://10.xxx.xxx.xxx:9093/#/status
- 17. Auto install & config exporters using Ansible 17 1. Install Ansible On iOS $brew install ansible On Linux $yum install ansible
- 18. 1818 2. Consist of Directory & Files - /etc/hosts - ~/ansible_hosts - ~/ansible/playbooks/prometheus_exporter.yml - ~/ansible/playbooks/files/prometheus/node_exporter-0.12.0.linux-amd64.tar.gz - ~/ansible/playbooks/files/prometheus/mysqld_exporter-0.8.1.linux-amd64.tar.gz - ~/ansible/playbooks/files/prometheus/start_node_exporter.sh - ~/ansible/playbooks/files/prometheus/start_mysqld_exporter.sh - ~/ansible/playbooks/files/prometheus/.my.cnf - ~/ansible/playbooks/files/prometheus/prometheus.yml.temp Auto install & config exporters using Ansible
- 19. 19 /etc/hosts ~/ansible_hosts #DB server lists 10.xxx.xxx.xxx dbhost01 10.xxx.xxx.xxx dbhost02 10.xxx.xxx.xxx dbhost03 [Group1] dbhost01 dbhost02 dbhost03 system hosts file ansible hosts file You should set following information on profile. export ANSIBLE_INVENTORY=~/ansible_hosts Auto install & config exporters using Ansible
- 20. 20 ~/ansible/playbooks/prometheus_exporter.yml - name: Prometheus Install and Configuration user : user1 hosts: Group1 sudo: yes tasks: - name: make directory file: path=/opt/prometheus_exporters/ state=directory - name: Copy tar.gz files copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/{{ item }} with_items: - node_exporter-0.12.0.linux-amd64.tar.gz - mysqld_exporter-0.8.1.linux-amd64.tar.gz continue next page… ansible playbook file for install and config prometheus exporter Auto install & config exporters using Ansible
- 21. 21 ~/ansible/playbooks/prometheus_exporter.yml - name: Uncompress gz files command: tar -xzvf /opt/prometheus_exporters/{{ item }} -C /opt/prometheus_exporters with_items: - node_exporter-0.12.0.linux-amd64.tar.gz - mysqld_exporter-0.8.1.linux-amd64.tar.gz - name: Rename node_exporter Directory name command: mv /opt/prometheus_exporters/node_exporter-0.12.0.linux-amd64 /opt/prometheus_exporters/node_exporter - name: Rename mysqld_exporter Directory name command: mv /opt/prometheus_exporters/mysqld_exporter-0.8.1.linux-amd64 /opt/prometheus_exporters/mysqld_exporter - name: change directory owner file: path=/opt/prometheus_exporters/{{ item }} state=directory owner=root group=root mode=640 with_items: - node_exporter - mysqld_exporter continue next page… Auto install & config exporters using Ansible
- 22. 22 ~/ansible/playbooks/prometheus_exporter.yml - name: Copy start_mysqld_exporter.sh copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/mysqld_exporter with_items: - start_mysqld_exporter.sh - name: Copy start_node_exporter.sh copy: src=files/prometheus/{{ item }} dest=/opt/prometheus_exporters/node_exporter with_items: - start_node_exporter.sh - name: .my.cnf coyp template: src=files/prometheus/.my.cnf dest=/opt/prometheus_exporters/mysqld_exporter owner=root group=root mode=640 Auto install & config exporters using Ansible continue next page…
- 23. 23 ~/ansible/playbooks/prometheus_exporter.yml - name: .my.cnf coyp template: src=files/prometheus/.my.cnf dest=/opt/prometheus_exporters/mysqld_exporter owner=root group=root mode=640 - name: Execute node exporter command: /bin/sh /opt/prometheus_exporters/node_exporter/start_node_exporter.sh - name: Execute mysql exporter command: /bin/sh /opt/prometheus_exporters/mysqld_exporter/start_mysqld_exporter.sh - name: move new host info on prometheus server template: src=files/prometheus/prometheus.yml.temp dest=/opt/prometheus/prometheus.yml.{{ansible_hostname}} owner=root group=root mode=640 delegate_to: monitoring_server_hostname - name: add new host info on prometheus.yml shell: cat /opt/prometheus/prometheus.yml.{{ansible_hostname}} >> /opt/prometheus/prometheus.yml delegate_to: monitoring_server_hostname Auto install & config exporters using Ansible
- 24. 24 ~/ansible/playbooks/files/prometheus/ node_exporter-0.12.0.linux-amd64.tar.gz mysqld_exporter-0.8.1.linux-amd64.tar.gz start_node_exporter.sh start_mysqld_exporter.sh : node_exporter install file : mysqld_exporter install file : node_exporter start script nohup /opt/prometheus_exporters/node_exporter/node_exporter & nohup /opt/prometheus_exporters/mysqld_exporter/mysqld_exporter -config.my-cnf="/opt/prometheus_exporters/mysqld_exporter/.my.cnf" & Auto install & config exporters using Ansible : mysqld_exporter start script
- 25. 25 ~/ansible/playbooks/files/prometheus/ .my.cnf [client] host= {{ansible_default_ipv4.address}} user=mon_user # db user for monitoring password=mon_user_passwd # db user password Auto install & config exporters using Ansible : mysqld_exporter config file mysql> GRANT REPLICATION CLIENT, PROCESS ON *.* TO ‘mon_user'@'127.0.0.1' identified by ‘mon_user_passwd'; mysql> GRANT SELECT ON performance_schema.* TO ‘mon_user'@'127.0.0.1'; You have to add db user for monitoring on db server
- 26. 26 ~/ansible/playbooks/files/prometheus/ prometheus.yml.temp - job_name: linux_{{ansible_hostname}} target_groups: - targets: ['{{ansible_default_ipv4.address}}:9100'] labels: alias: {{ansible_hostname}} - job_name: mysql_{{ansible_hostname}} target_groups: - targets: ['{{ansible_default_ipv4.address}}:9104'] labels: alias: {{ansible_hostname}} Auto install & config exporters using Ansible : prometheus config information
- 27. Install & config grafana on monitoring server 27 1. Install Grafana Download rpm file http://grafana.org/download/ #rpm –ivh grafana-3.1.1-1470047149.x86_64.rpm #/etc/init.d/grafana start 2. Grafana connect on web browser http://10.xxx.xxx.xxx:3000/
- 28. 28 Make grafana dashboard CPU Usage Disk Bandwidth Disk space MySQL Active Threads Load Average MySQL QPS Network Traffic Memory
- 29. Make grafana dashboard 29 1. Register Datasource
- 30. Make grafana dashboard 30 2. Create new dashboard and templating
- 31. 31 Make grafana dashboard 3. Create a templating
- 32. 32 Make grafana dashboard 4. Add Graph-Load average(1)
- 33. 33 Make grafana dashboard 4. Add Graph-Load average(2)
- 34. 34 Make grafana dashboard 4. Add Graph-Load average(3)
- 35. 35 Make grafana dashboard 4. Add Graph-Load average(4)
- 36. 36 Make grafana dashboard 4. Add Graph-Load average(5)
- 37. 37 Make grafana dashboard 4. Add Graph-Load average(6)
- 38. 38 Make grafana dashboard 4. Add Graph-Load average(7)
- 39. 39 Make grafana dashboard 5. Add Graph-CPU Usage 6. Add Graph-Disk Bandwidth
- 40. 40 Make grafana dashboard 7. Add Graph-Disk space Query Detail ((node_filesystem_size{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"} - node_filesystem_avail{fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"}) /node_filesystem_size{ fstype!~"rootfs|selinuxfs|autofs|rpc_pipefs|tmpfs",alias="$host"}) * 100
- 41. 41 Make grafana dashboard 8. Add Graph-MySQL Active Threads 9. Add Graph-MySQL QPS
- 42. 42 Make grafana dashboard 10. Add Graph-Network Traffic
- 43. 43 Make grafana dashboard 11. Add Graph-Memory
- 44. 44 Make grafana dashboard You can get more various kinds of dashboard. https://github.com/percona/grafana-dashboards
- 45. Thank You