SlideShare a Scribd company logo
using Beats & ELK
MySQL Slow Query log Monitoring
About me
2
dba.kim@gmail.com
Architecture
MySQL Slow Log
DB Servers ELK Server
Logstash Elasticsearch
FileBeat Kibana
3
Install & Config FileBeat
4
# rpm -ivh filebeat-1.0.1-x86_64.rpm
$vi /etc/filebeat/filebeat.yml
filebeat:
prospectors:
paths:
- /db/data01/mysql-slow.log //slow query path
output:
#elasticsearch: //comment
#hosts: ["localhost:9200"] //comment
logstash: //uncomment
# The Logstash hosts
hosts: ["10.xx.xx.xx:5044"] //logstash server ip
1. Install FileBeat – on DB servers
2. Parameter configuration
5
# /etc/init.d/filebeat start
Starting filebeat: [ OK ]
3. Start FileBeat – on DB servers
Install & Config FileBeat
Install & Config Elasticsearch
6
# tar –xzvf elasticsearch-2.1.1.tar.gz
1. Install Elasticsearch – on ELK servers
2. configuration
$ vi ./elasticsearch-2.1.1/config/elasticsearch.yml
cluster.name : log_cluster # cluster name
node.name : slow_log # node name
path.data: /DATA/data # index data path
path.logs: /DATA/logs # log path
network.host : 10.xxx.xxx.xxx # server’s ip
7
3. start elasticsearch
$./bin/elasticsearch
You can’t run elastisearch as root.
Install & Config Elasticsearch
Install & Config Logstash
8
1. Install Logstash – on ELK server
$ rpm –ivh logstash-2.1.1-1.noarch.rpm
2-1. Configure(input plugin)
# vi /etc/logstash/conf.d/10-slow-log.conf
input {
beats {
port => 5044
codec => multiline{
pattern => "^# Time:"
negate => true
what => previous
}
}
}
9
2-2. Configure(filter plugin)
filter {
grok {
match => [ "message", "^# User@Host: %{USER:query_user}(?:[[^]]+])?s+@s+%{HOSTNAME:query_host}?s+[%{IP:query_ip}?]" ]
}
grok {
match => [ "message", "^#
Thread_id: %{NUMBER:thread_id:int}s+Schema: %{USER:schema}s+Last_errno: %{NUMBER:last_errno:int}s+Killed: %{NUMBER:killed:int}"]
}
grok {
match => [ "message", "^# Query_time: %{NUMBER:query_time:float}s+Lock_time: %{NUMBER:lock_time}s+ Rows_sent: %{NUMBER:rows_sent:int}
s+Rows_examined: %{NUMBER:rows_examined:int}s+Rows_affected: %{NUMBER:rows_affected:int}s+Rows_read: %{NUMBER:rows_read:int}"]
}
grok { match => [ "message", "^# Bytes_sent: %{NUMBER:bytes_sent:float}"] }
grok { match => [ "message", "^SET timestamp=%{NUMBER:timestamp}" ] }
grok { match => [ "message", "^SET timestamp=%{NUMBER};s+%{GREEDYDATA:query}" ] }
date { match => [ "timestamp", "UNIX" ] }
mutate {
remove_field => "timestamp"
}
}
Install & Config Logstash
10
2-3. Configure(output plugin)
output {
elasticsearch {
hosts => "10.xx.xx.xx"
}
}
3. Beats plugin install
# cd /opt/logstash/bin
# ./plugin install logstash-input-beats
4. Start logstash
# /etc/init.d/logstash start
Install & Config Logstash
Install & Config Kibana
11
1. Install Kibana – on ELK server
$ tar –xvf kibana-4.3.1-linux-x64.tar.gz
2. Configure
$ vi ./kibana-4.3.1-linux-x64/config/kibana.yml
server.host: "10.xx.xx.xx“ # kibana server ip
elasticsearch.url: "http://10.xx.xx.xx:9200" # elasticsearch server ip
3. Start Kibana
$ ./bin/kibana
Visualize – Slow query graph 12
mouse over
Install & Config Kibana
13
New Visualization > Line chart
Install & Config Kibana
Visualize – Slow query graph
Dashboard - sample 14
http://10.xxx.xxx.xxx:5601
Install & Config Kibana
Thank You

More Related Content

PPTX
MySQL Audit using Percona audit plugin and ELK
PDF
MySQL Audit using Percona audit plugin and ELK
PPTX
ProxySQL & PXC(Query routing and Failover Test)
PPTX
My sql failover test using orchestrator
PPTX
MySQL Monitoring using Prometheus & Grafana
PPTX
Query logging with proxysql
PDF
Cassandra summit 2013 - DataStax Java Driver Unleashed!
PDF
glance replicator
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELK
ProxySQL & PXC(Query routing and Failover Test)
My sql failover test using orchestrator
MySQL Monitoring using Prometheus & Grafana
Query logging with proxysql
Cassandra summit 2013 - DataStax Java Driver Unleashed!
glance replicator

What's hot (20)

PDF
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
PPT
Replica Sets (NYC NoSQL Meetup)
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
PDF
MongoDB Database Replication
PPT
Why and How Powershell will rule the Command Line - Barcamp LA 4
PDF
How to Run Solr on Docker and Why
PDF
Failsafe Mechanism for Yahoo Homepage
PDF
[오픈소스컨설팅] 프로메테우스 모니터링 살펴보고 구성하기
PDF
MySQL replication & cluster
PPTX
Administering and Monitoring SolrCloud Clusters
PDF
Replacing Squid with ATS
PPTX
HADOOP 실제 구성 사례, Multi-Node 구성
PDF
Elasticsearch for Logs & Metrics - a deep dive
PDF
Tuning Solr & Pipeline for Logs
PDF
Tuning Solr for Logs
PDF
Set up Hadoop Cluster on Amazon EC2
PDF
Docker Monitoring Webinar
PPTX
Introduction to PostgreSQL
PDF
Docker Setting for Static IP allocation
PDF
Solr for Indexing and Searching Logs
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
Replica Sets (NYC NoSQL Meetup)
Solr Search Engine: Optimize Is (Not) Bad for You
MongoDB Database Replication
Why and How Powershell will rule the Command Line - Barcamp LA 4
How to Run Solr on Docker and Why
Failsafe Mechanism for Yahoo Homepage
[오픈소스컨설팅] 프로메테우스 모니터링 살펴보고 구성하기
MySQL replication & cluster
Administering and Monitoring SolrCloud Clusters
Replacing Squid with ATS
HADOOP 실제 구성 사례, Multi-Node 구성
Elasticsearch for Logs & Metrics - a deep dive
Tuning Solr & Pipeline for Logs
Tuning Solr for Logs
Set up Hadoop Cluster on Amazon EC2
Docker Monitoring Webinar
Introduction to PostgreSQL
Docker Setting for Static IP allocation
Solr for Indexing and Searching Logs
Ad

Similar to MySQL Slow Query log Monitoring using Beats & ELK (20)

PPT
Elk presentation 2#3
PDF
Introducing ELK
PPTX
Elk ruminating on logs
PDF
Brisbane DevOps Meetup - Logstash
PDF
Log analysis with the elk stack
PPT
ELK stack at weibo.com
PPTX
Elastic stack Presentation
PPTX
ELK Ruminating on Logs (Zendcon 2016)
PPT
Logstash
PDF
LogStash in action
PDF
OSDC 2016 - Unifying Logs and Metrics Data with Elastic Beats by Monica Sarbu
PPTX
Log management with ELK
PDF
ELK: a log management framework
PPTX
Kibana+ElasticSearch+LogStash to handle Log messages on Prod servers
PDF
"How about no grep and zabbix?". ELK based alerts and metrics.
KEY
Zero mq logs
PDF
Log analysis with elastic stack
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
ODP
Log aggregation and analysis
PDF
Null Bachaav - May 07 Attack Monitoring workshop.
Elk presentation 2#3
Introducing ELK
Elk ruminating on logs
Brisbane DevOps Meetup - Logstash
Log analysis with the elk stack
ELK stack at weibo.com
Elastic stack Presentation
ELK Ruminating on Logs (Zendcon 2016)
Logstash
LogStash in action
OSDC 2016 - Unifying Logs and Metrics Data with Elastic Beats by Monica Sarbu
Log management with ELK
ELK: a log management framework
Kibana+ElasticSearch+LogStash to handle Log messages on Prod servers
"How about no grep and zabbix?". ELK based alerts and metrics.
Zero mq logs
Log analysis with elastic stack
Attack monitoring using ElasticSearch Logstash and Kibana
Log aggregation and analysis
Null Bachaav - May 07 Attack Monitoring workshop.
Ad

More from I Goo Lee (20)

PDF
MySQL_Fabric_운영시유의사항
PDF
MySQL Deep dive with FusionIO
PDF
From MSSQL to MySQL
PDF
From MSSQL to MariaDB
PDF
AWS Aurora 100% 활용하기
PDF
Backup automation in KAKAO
PDF
텔레그램을 이용한 양방향 모니터링 시스템 구축
PDF
Federated Engine 실무적용사례
PDF
MySQL 상태 메시지 분석 및 활용
PDF
MySQL 5.7 NF – Optimizer Improvement
PDF
MySQL 5.7 NF – JSON Datatype 활용
PDF
Intro KaKao MRTE (MySQL Realtime Traffic Emulator)
PDF
MS 빅데이터 서비스 및 게임사 PoC 사례 소개
PDF
AWS 환경에서 MySQL Infra 설계하기-2본론
PDF
AWS 환경에서 MySQL Infra 설계하기-1도입부분
PDF
AWS 환경에서 MySQL BMT
PDF
PostgreSQL 이야기
PDF
Intro KaKao ADT (Almighty Data Transmitter)
PDF
Binlog Servers 구축사례
PDF
Intro ProxySQL
MySQL_Fabric_운영시유의사항
MySQL Deep dive with FusionIO
From MSSQL to MySQL
From MSSQL to MariaDB
AWS Aurora 100% 활용하기
Backup automation in KAKAO
텔레그램을 이용한 양방향 모니터링 시스템 구축
Federated Engine 실무적용사례
MySQL 상태 메시지 분석 및 활용
MySQL 5.7 NF – Optimizer Improvement
MySQL 5.7 NF – JSON Datatype 활용
Intro KaKao MRTE (MySQL Realtime Traffic Emulator)
MS 빅데이터 서비스 및 게임사 PoC 사례 소개
AWS 환경에서 MySQL Infra 설계하기-2본론
AWS 환경에서 MySQL Infra 설계하기-1도입부분
AWS 환경에서 MySQL BMT
PostgreSQL 이야기
Intro KaKao ADT (Almighty Data Transmitter)
Binlog Servers 구축사례
Intro ProxySQL

Recently uploaded (20)

PDF
Unit-1 introduction to cyber security discuss about how to secure a system
PPTX
Introduction about ICD -10 and ICD11 on 5.8.25.pptx
PDF
RPKI Status Update, presented by Makito Lay at IDNOG 10
PPTX
artificial intelligence overview of it and more
PDF
💰 𝐔𝐊𝐓𝐈 𝐊𝐄𝐌𝐄𝐍𝐀𝐍𝐆𝐀𝐍 𝐊𝐈𝐏𝐄𝐑𝟒𝐃 𝐇𝐀𝐑𝐈 𝐈𝐍𝐈 𝟐𝟎𝟐𝟓 💰
PPTX
SAP Ariba Sourcing PPT for learning material
PPTX
Internet___Basics___Styled_ presentation
PDF
An introduction to the IFRS (ISSB) Stndards.pdf
PDF
The New Creative Director: How AI Tools for Social Media Content Creation Are...
PPTX
522797556-Unit-2-Temperature-measurement-1-1.pptx
PDF
Slides PDF The World Game (s) Eco Economic Epochs.pdf
PDF
Paper PDF World Game (s) Great Redesign.pdf
PDF
WebRTC in SignalWire - troubleshooting media negotiation
PPTX
Job_Card_System_Styled_lorem_ipsum_.pptx
PPTX
Introuction about WHO-FIC in ICD-10.pptx
PDF
Tenda Login Guide: Access Your Router in 5 Easy Steps
PDF
Cloud-Scale Log Monitoring _ Datadog.pdf
PPTX
Funds Management Learning Material for Beg
PPTX
June-4-Sermon-Powerpoint.pptx USE THIS FOR YOUR MOTIVATION
PPTX
QR Codes Qr codecodecodecodecocodedecodecode
Unit-1 introduction to cyber security discuss about how to secure a system
Introduction about ICD -10 and ICD11 on 5.8.25.pptx
RPKI Status Update, presented by Makito Lay at IDNOG 10
artificial intelligence overview of it and more
💰 𝐔𝐊𝐓𝐈 𝐊𝐄𝐌𝐄𝐍𝐀𝐍𝐆𝐀𝐍 𝐊𝐈𝐏𝐄𝐑𝟒𝐃 𝐇𝐀𝐑𝐈 𝐈𝐍𝐈 𝟐𝟎𝟐𝟓 💰
SAP Ariba Sourcing PPT for learning material
Internet___Basics___Styled_ presentation
An introduction to the IFRS (ISSB) Stndards.pdf
The New Creative Director: How AI Tools for Social Media Content Creation Are...
522797556-Unit-2-Temperature-measurement-1-1.pptx
Slides PDF The World Game (s) Eco Economic Epochs.pdf
Paper PDF World Game (s) Great Redesign.pdf
WebRTC in SignalWire - troubleshooting media negotiation
Job_Card_System_Styled_lorem_ipsum_.pptx
Introuction about WHO-FIC in ICD-10.pptx
Tenda Login Guide: Access Your Router in 5 Easy Steps
Cloud-Scale Log Monitoring _ Datadog.pdf
Funds Management Learning Material for Beg
June-4-Sermon-Powerpoint.pptx USE THIS FOR YOUR MOTIVATION
QR Codes Qr codecodecodecodecocodedecodecode

MySQL Slow Query log Monitoring using Beats & ELK

  • 1. using Beats & ELK MySQL Slow Query log Monitoring
  • 3. Architecture MySQL Slow Log DB Servers ELK Server Logstash Elasticsearch FileBeat Kibana 3
  • 4. Install & Config FileBeat 4 # rpm -ivh filebeat-1.0.1-x86_64.rpm $vi /etc/filebeat/filebeat.yml filebeat: prospectors: paths: - /db/data01/mysql-slow.log //slow query path output: #elasticsearch: //comment #hosts: ["localhost:9200"] //comment logstash: //uncomment # The Logstash hosts hosts: ["10.xx.xx.xx:5044"] //logstash server ip 1. Install FileBeat – on DB servers 2. Parameter configuration
  • 5. 5 # /etc/init.d/filebeat start Starting filebeat: [ OK ] 3. Start FileBeat – on DB servers Install & Config FileBeat
  • 6. Install & Config Elasticsearch 6 # tar –xzvf elasticsearch-2.1.1.tar.gz 1. Install Elasticsearch – on ELK servers 2. configuration $ vi ./elasticsearch-2.1.1/config/elasticsearch.yml cluster.name : log_cluster # cluster name node.name : slow_log # node name path.data: /DATA/data # index data path path.logs: /DATA/logs # log path network.host : 10.xxx.xxx.xxx # server’s ip
  • 7. 7 3. start elasticsearch $./bin/elasticsearch You can’t run elastisearch as root. Install & Config Elasticsearch
  • 8. Install & Config Logstash 8 1. Install Logstash – on ELK server $ rpm –ivh logstash-2.1.1-1.noarch.rpm 2-1. Configure(input plugin) # vi /etc/logstash/conf.d/10-slow-log.conf input { beats { port => 5044 codec => multiline{ pattern => "^# Time:" negate => true what => previous } } }
  • 9. 9 2-2. Configure(filter plugin) filter { grok { match => [ "message", "^# User@Host: %{USER:query_user}(?:[[^]]+])?s+@s+%{HOSTNAME:query_host}?s+[%{IP:query_ip}?]" ] } grok { match => [ "message", "^# Thread_id: %{NUMBER:thread_id:int}s+Schema: %{USER:schema}s+Last_errno: %{NUMBER:last_errno:int}s+Killed: %{NUMBER:killed:int}"] } grok { match => [ "message", "^# Query_time: %{NUMBER:query_time:float}s+Lock_time: %{NUMBER:lock_time}s+ Rows_sent: %{NUMBER:rows_sent:int} s+Rows_examined: %{NUMBER:rows_examined:int}s+Rows_affected: %{NUMBER:rows_affected:int}s+Rows_read: %{NUMBER:rows_read:int}"] } grok { match => [ "message", "^# Bytes_sent: %{NUMBER:bytes_sent:float}"] } grok { match => [ "message", "^SET timestamp=%{NUMBER:timestamp}" ] } grok { match => [ "message", "^SET timestamp=%{NUMBER};s+%{GREEDYDATA:query}" ] } date { match => [ "timestamp", "UNIX" ] } mutate { remove_field => "timestamp" } } Install & Config Logstash
  • 10. 10 2-3. Configure(output plugin) output { elasticsearch { hosts => "10.xx.xx.xx" } } 3. Beats plugin install # cd /opt/logstash/bin # ./plugin install logstash-input-beats 4. Start logstash # /etc/init.d/logstash start Install & Config Logstash
  • 11. Install & Config Kibana 11 1. Install Kibana – on ELK server $ tar –xvf kibana-4.3.1-linux-x64.tar.gz 2. Configure $ vi ./kibana-4.3.1-linux-x64/config/kibana.yml server.host: "10.xx.xx.xx“ # kibana server ip elasticsearch.url: "http://10.xx.xx.xx:9200" # elasticsearch server ip 3. Start Kibana $ ./bin/kibana
  • 12. Visualize – Slow query graph 12 mouse over Install & Config Kibana
  • 13. 13 New Visualization > Line chart Install & Config Kibana Visualize – Slow query graph
  • 14. Dashboard - sample 14 http://10.xxx.xxx.xxx:5601 Install & Config Kibana