#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
0 likes32 views
The document discusses improving database observability. It recommends collecting metrics like utilization, saturation, errors, latency, traffic, and errors using methods like USE, RED, and the seven golden signals. These metrics can be collected via tools like Collectd and exported to dashboards. Logs of SQL queries and slow queries should also be collected and analyzed. The goal is to increase transparency and visibility of the database to help users and developers and improve monitoring and reliability. Future work includes enhancing SQL logging, using sys_schema, publishing SLAs, and open sourcing monitoring probes.
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
- 1. How to improve database observability? @Charles_JUDITH Paris Open Source Summit 2019
- 2. About me ● Senior Site Reliability Engineer at Criteo ● Working on monitoring topics since few years ● Currently providing the (open source) database service at Criteo ● @Charles_JUDITH on Twitter
- 3. Agenda 1. Context 2. First iteration 3. Second iteration 4. Next steps 5. Resources
- 4. Context
- 5. Goal ● Alerting ● No hidden issues ● An observable platform! ● The DBA team shouldn’t be a “blocker” for the users!
- 7. OBSERVABILITY IS A MEASURE OF HOW WELL INTERNAL STATES OF A SYSTEM CAN BE INFERRED FROM KNOWLEDGE OF ITS EXTERNAL OUTPUTS. » SOURCE: WIKIPEDIA
- 8. My opinion about observability ● It’s not only about the tools ● It’s not a fancy name to say “monitoring” ● It’s more about “transparency”
- 9. Why a system needs to be observable?
- 10. Why a system needs to be observable? ● Is it working as expected by the users? ● To answer basic questions about your service/platform ● Increase the visibility for you and your users/customers ● Long term tends analysis ● “If can’t measure it, you can’t manage it”
- 11. Observability is fundamental for reliability Analogy to the Maslow’s hierarchy of needs
- 13. The observability effects ● Giving superpowers ● It’s like a roller coaster ● You need to be patient
- 14. Let’s go!
- 15. Metrics
- 16. How to start?
- 17. USE method ● USE was introduced by @brendangregg ● Utilization: disk,CPU usage … ● Saturation: disk I/O ● Errors: network interface errors
- 18. The four golden signals ● Introduced in the Google SRE book ● Latency: response time, queue/wait time ● Traffic: A measure of how much demand is being placed on the service ● Errors: The rate of requests that fail ● Saturation: How “full” is the service
- 19. RED method ● RED was introduced by @tom_wilkie ● (Request) Rate - the number of requests, per second, you services are serving. ● (Request) Errors - the number of failed requests per second. ● (Request) Duration - distributions of the amount of time each request takes. ● Subset of “The Four Golden Signals”
- 20. The seven golden signals ● CELT + USE introduced by @xaprb ● Concurrency: number of simultaneous requests ● Error rate ● Latency: response time ● Throughput: query per seconds (QPS)
- 21. CASE method ● CASE was introduced by @gphat ● Context-heavy ● Actionnable ● Symptom-based ● Evaluated
- 22. Preferred approach ● The seven golden signals ● Good to measure the service quality ● System and application metrics are valuable in our case
- 23. How to collect the metrics? ● Collectd ● Node exporter ● MySQLD exporter ● Python MySQL plugin for CollectD ● Few others
- 24. What to do with all these metrics? ● Pick some useful “indicators” like: ○ thread usage ○ service status ○ backup status, duration, size ○ replication lag
- 25. How to show/use those metrics?
- 26. Global overview
- 27. InnoDB metrics
- 28. Simple user view
- 29. USE dashboard
- 30. Disk partition full with tmp_table
- 32. Database cleaning and optimize table
- 33. DATABASES EXPOSE LOTS OF METRICS ABOUT THEIR STATUS, BUT MUCH LESS ABOUT THE DETAILS OF THEIR WORKLOAD.
- 34. “WE THINK OUR DATABASE IS SLOW?” “Last week week we noticed that the database was slow.”
- 36. Logs
- 37. Logs ● Logs all the SQL queries (general log) ● Install an agent to ship those logs with “custom fields” ● Make the logs available for our users
- 38. Logs ● Logs all the SQL queries (general log) ● Install an agent to ship those logs with “custom fields” ● Configure MySQL/MariaDB to log the slow queries ● Use Rsyslog with a custom template! ● Make the logs available for our users
- 40. Conclusions ● The DBA is not a blocker for the developers ● The visibility and transparency on the database service ● Happy customers/developers/users ● Effective monitoring ● Shipping slow queries is not easy ● In that case metrics and logs is a good combo but we want more!
- 41. Next steps ● Continue to improve the SQL logging ● Leverage the usage of sys_schema ● Metrics per database ● Publish the SLA ● Open source our probe for MySQL/MariaDB
- 43. Thank you!