![Golang meetup # 11
Metrics and labels
Metric types:
- counter
- gauge
- histogram
- summary
var httpResponsesTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "my_super_puper_production",
Subsystem: "http_server",
Name: "http_responses_total",
Help: "The count of http responses issued, classified by
code and method.",
},
[]string{"code", "method"},
)
Listing 1
CAUTION: Remember that every unique combination of key-
value label pairs represents a new time series, which can
dramatically increase the amount of data stored. Do not use
labels to store dimensions with high cardinality (many different
label values), such as user IDs, email addresses, or other
unbounded sets of values.](https://image.slidesharecdn.com/monitoring-at-lazada-170606082516/85/GO-Monitoring-at-lazada-8-320.jpg)
![Golang meetup # 11
Powerful query language
topk(3, sum(rate(bazooka_instance_cpu_time_ns[5m])) by (app, proc))
sort_desc(sum(bazooka_instance_memory_limit_bytes -
bazooka_instance_memory_usage_bytes) by (app, proc))
Expression browser](https://image.slidesharecdn.com/monitoring-at-lazada-170606082516/85/GO-Monitoring-at-lazada-9-320.jpg)
![Golang meetup # 11
What do we monitor for our Review API?
- CPU / memory
- Goroutines
- Avg RT, RT per handler
- Error rates
- Throughput [per clients]
- DB connection
- Cache size (number of items)
- Cache hits / misses (utilization)
- …](https://image.slidesharecdn.com/monitoring-at-lazada-170606082516/85/GO-Monitoring-at-lazada-17-320.jpg)
GO Monitoring at lazada
- 1. GO Monitoring at Lazada Rinat Takhautdinov & Ninh Dang Senior Software Engineers at PDP Vertical Team – Lazada
- 2. Golang meetup # 11 About Us – PDP Vertical Department - We’re doing product development at Lazada - Product Detail Page - Home Page - Login/Signup - Work mostly with Golang and PHP
- 3. Golang meetup # 11 - Overview of time-series metrics monitoring - Big picture - Prometheus - Grafana - How monitoring helps us at Lazada - Use case 1: Product Ratings & Reviews performance optimization - Use case 2: Shaping the product for Buyer-Seller Communication - Conclusions Agenda
- 4. Golang meetup # 11 - Overview of time-series metrics monitoring - Big picture - Prometheus - Grafana - How monitoring helps us at Lazada - Use case 1: Product Ratings & Reviews performance optimization - Use case 2: Shaping the product for Buyer-Seller Communication - Conclusions Agenda
- 5. Golang meetup # 11 Monitoring: General Architecture http://service:port/metrics
- 6. Golang meetup # 11 What does Prometheus do? • Time series database • Written in Go - no external dependencies • Powerful functional query language • Pull, not push • Thousands of targets • Hundreds of thousands of samples per second • Millions of time series • Distributed (or not) a) can be run as a single node b) can be run as a cluster for redundancy & performance
- 7. Golang meetup # 11 Exporters and direct instrumentation Client libraries https://prometheus.io/docs/instrumenting/clientlibs/ Official Go (https://github.com/prometheus/client_golang) Java (JVM) Ruby Python Unofficial .NET / C# Node.js Bash Exporters https://prometheus.io/docs/instrumenting/exporters/ Official Node/system metrics exporter Graphite exporter Collectd exporter JMX exporter HAProxy exporter StatsD bridge AWS CloudWatch exporter Hystrix metrics publisher Mesos task exporter Consul exporter Direct instrumentation Kubernetes Kubernetes-Mesos Etcd gokit go-metrics instrumentation library
- 8. Golang meetup # 11 Metrics and labels Metric types: - counter - gauge - histogram - summary var httpResponsesTotal = prometheus.NewCounterVec( prometheus.CounterOpts{ Namespace: "my_super_puper_production", Subsystem: "http_server", Name: "http_responses_total", Help: "The count of http responses issued, classified by code and method.", }, []string{"code", "method"}, ) Listing 1 CAUTION: Remember that every unique combination of key- value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.
- 9. Golang meetup # 11 Powerful query language topk(3, sum(rate(bazooka_instance_cpu_time_ns[5m])) by (app, proc)) sort_desc(sum(bazooka_instance_memory_limit_bytes - bazooka_instance_memory_usage_bytes) by (app, proc)) Expression browser
- 10. Golang meetup # 11 Grafana
- 11. Golang meetup # 11 Grafana
- 12. Golang meetup # 11 Grafana
- 13. Golang meetup # 11 - Overview of time-series metrics monitoring - Big picture - Prometheus - Grafana - How monitoring helps us at Lazada - Use case 1: Product Ratings & Reviews performance optimization - Use case 2: Shaping the product for Buyer-Seller Communication - Conclusions Agenda
- 14. Golang meetup # 11 Product Ratings & Reviews
- 15. Golang meetup # 11 Problem – High level - Lazada has many campaigns throughout the year - How do we prepare for the campaigns? - Stress tests + monitoring - Find & fix bottlenecks - Verify results - What happened to us? - Site was broken, checkout not working - Backend service alert (Review API)
- 16. Golang meetup # 11 Review API architecture
- 17. Golang meetup # 11 What do we monitor for our Review API? - CPU / memory - Goroutines - Avg RT, RT per handler - Error rates - Throughput [per clients] - DB connection - Cache size (number of items) - Cache hits / misses (utilization) - …
- 18. Golang meetup # 11 What do we monitor for our Review API?
- 19. Golang meetup # 11 What do we monitor for our Review API?
- 20. Golang meetup # 11 What we found in our case? - Number of cache items went up very high - Review API version fragmentation - Many calls on old version API endpoints
- 21. Golang meetup # 11 Optimization in action 1. Reduce number of calls from clients. 2. Force all clients move to use the latest versions. 3. Deprecate outdated handlers (keep multi-version of API is expensive)
- 22. Golang meetup # 11 Feel rock?
- 23. Golang meetup # 11 Business metrics monitoring for product development Product delivering focused team Many kind of KPIs how to check product health need to tracked A lot of ideas how to make our products better Limited capacity to do it Smooth rollout and looking for better way
- 24. Golang meetup # 11 Business metrics monitoring for product development
- 25. Golang meetup # 11 Business metrics monitoring for product development - Helps us to validate and evolve our MVP - Transparent for everyone (e.g. dev team, country PMs) - Real time tracking product health
- 26. Golang meetup # 11 Conclusions - Having monitoring is extremely critical for any business - If you have many teams, having standard is important - Define most important metrics for each microservice - Have some TV screens with some graphs near your team corner!
- 27. Golang meetup # 11 Q&A
- 28. Golang meetup # 11 - WE ARE HIRING! - 4 x Senior Backend Developers (GO/PHP) - Contact us for more information: - ninh.dang@lazada.com - rinat.takhautdinov@lazada.com
Editor's Notes
- #23: -rps:1k