Monitoring kubernetes with prometheus
8 likes1,270 views
The document discusses monitoring microservices using Prometheus in Kubernetes, including setting up a local cluster with Minikube, deploying applications, and implementing monitoring tools. It covers the monitoring maturity model, types of monitoring (whitebox vs blackbox), and the importance of key metrics for continuous improvement. Additionally, it details practical steps for deploying Prometheus and Grafana for infrastructure monitoring, along with examples of using PromQL for metric queries.
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-150-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar)
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-151-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar)
Churn Ratemonth = ΔCcancel
C × Δt
t0, t1, t2, t3, t4, t5, t6](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-152-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-153-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-154-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Churn Ratemonth = ΔCcancel
C × Δt
t0, t1, t2, t3, t4, t5, t6
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-155-320.jpg)
![Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Built-in rate function
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-156-320.jpg)
![Worked
Example:
Churn rate
Churn Ratemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-160-320.jpg)
![Worked
Example:
Churn rate
Churn Ratemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))
Churn Ratemonth = ΔCcancel
C × Δt](https://image.slidesharecdn.com/monitoringkuberneteswithprometheus-170926145706/85/Monitoring-kubernetes-with-prometheus-161-320.jpg)
Monitoring kubernetes with prometheus
- 1. Making sure your containers aren’t on fire Monitoring microservices with Prometheus Brice Fernandes @fractallambda
- 3. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 4. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 7. How I.T. Was OS App Foo v1.1.0 Foo v1.5.0
- 8. How I.T. Was OS App Foo v1.1.0 Foo v1.5.0 ?
- 10. How I.T. Was Reproducible Deployment Continuous Deployment Fault Recovery Memory & CPU allocation Managing VMs? ? ? ? ? ? ?
- 14. Reproducible deployments Fault recovery Continuous deployment Don’t care about machine virtualisation Memory & CPU multiplexing Buzzword compliance The New Hotness
- 15. But…
- 16. But…
- 17. But…
- 18. But…
- 19. But…
- 21. Kubernetes – Greek for Helmsman or Pilot
- 28. This is what I want
- 29. This is what I want
- 30. This is what I want
- 31. This is what I want
- 32. This is what I want xxx.xxx.xxx.xxx:30003
- 33. This is what I want xxx.xxx.xxx.xxx:30003
- 36. ➤ minikube start
- 37. ➤ minikube start Starting local Kubernetes v1.7.5 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components… Kubectl is now configured to use the cluster.
- 38. ➤ minikube start Starting local Kubernetes v1.7.5 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components… Kubectl is now configured to use the cluster. Start a local cluster
- 39. ➤ minikube start Starting local Kubernetes v1.7.5 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components… Kubectl is now configured to use the cluster. Set up the kubernetes tools to point to our cluster
- 40. ➤ kubectl get all
- 41. ➤ kubectl get all NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kubernetes 10.0.0.1 <none> 443/TCP 5m
- 42. ➤ kubectl get all NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kubernetes 10.0.0.1 <none> 443/TCP 5m Default kubernetes service
- 43. HOST
- 44. HOST VM
- 46. Deploy an app
- 47. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v1
- 48. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v1 Our definition manifest
- 50. Where to find the image
- 51. How many to run
- 52. Which port to expose externally
- 53. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v1 deployment "mighty-fine-fe" created service "mighty-fine-fe" created Creates our pods
- 54. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v1 deployment "mighty-fine-fe" created service "mighty-fine-fe" created Exposes a service
- 55. ➤ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) kubernetes 10.0.0.1 <none> 443/TCP mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP Port 3000 of app is visible on port 30001 of cluster
- 56. ➤ open http://$(minikube ip):30001
- 57. ➤ open http://$(minikube ip):30001
- 58. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 59. Why monitor?
- 61. NOT about collecting data Why vs How
- 62. IS relevant outside of IT
- 64. A: What’s the purpose of your organisation?
- 65. Maybe: Educational goals # People reached # Papers published
- 66. Probably £/$/€
- 67. Metrics come from purpose. Monitor your goals
- 70. Ignorance0 The Monitoring Ladder Availability Collection Aggregation Analysis 1 Learning Automation Proactivity 2 3 4 5 6 7 You know whether your systems are available. You may have alerts
- 71. Ignorance0 The Monitoring Ladder Availability Logging Aggregation Analysis 1 Learning Automation Proactivity 2 3 4 5 6 7 You collect logs. Forensics is possible . Alerts
- 72. Ignorance0 The Monitoring Ladder Availability Collection Aggregation Analysis 1 Learning Automation Proactivity 2 3 4 5 6 7 You aggregate and persist data in a central place. Correlation is possible. Alerts Logs Forensics
- 73. Ignorance0 The Monitoring Ladder Availability Collection Analysis 1 Learning Automation Proactivity 2 4 5 6 7 You actually analyse the aggregated and correlated data. Use it to fix issues. Alerts Logs Forensics Aggregation3Persistence
- 74. Ignorance0 The Monitoring Ladder Availability Collection Analysis 1 Learning Automation Proactivity 2 4 5 6 7 Root cause analysis. Strengthening fixes. Antifragile. Still responsive. Alerts Logs Forensics Aggregation3Persistence
- 75. Ignorance0 The Monitoring Ladder Availability Collection Analysis 1 Learning Automation Proactivity 2 4 5 6 7 Automated remedial actions. Data collection for analysis. No customer impact. Alerts Logs Forensics Aggregation3Persistence Antifragile
- 76. Ignorance0 The Monitoring Ladder Availability Collection Analysis 1 Learning Automation Proactivity 2 4 5 6 7 Automated remedial actions. Data collection for analysis. No customer impact. Alerts Logs Forensics Aggregation3Persistence Antifragile
- 77. Ignorance0 The Monitoring Ladder Availability Collection Analysis 1 Learning Automation Proactivity 2 4 5 6 7 Active strengthening by attacking production systems. Alerts Logs Forensics Aggregation3Persistence Antifragile 0-Impact
- 79. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 82. Push vs Pull
- 83. Push vs Pull
- 86. Which one is right? Pull or Push? Whitebox or Blackbox?
- 87. Which one is right? Pull or Push? Whitebox or Blackbox? Both
- 89. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 91. ➤ kubectl apply -f https://tinyurl.com/kube-prom-monitoring deployment "prometheus" created service "prometheus" created service "internal-prometheus" created deployment "grafana" created service "grafana" created configmap "prometheus-configmap" created Create and expose Prometheus
- 92. ➤ kubectl apply -f https://tinyurl.com/kube-prom-monitoring deployment "prometheus" created service "prometheus" created service "internal-prometheus" created deployment "grafana" created service "grafana" created configmap "prometheus-configmap" created Create and expose Grafana
- 93. ➤ kubectl apply -f https://tinyurl.com/kube-prom-monitoring deployment "prometheus" created service "prometheus" created service "internal-prometheus" created deployment "grafana" created service "grafana" created configmap "prometheus-configmap" created Configure Prometheus Using a ConfigMap
- 94. ➤ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) grafana 10.0.0.120 <nodes> 3000:30002/TCP internal-prometheus 10.0.0.39 <none> 9090/TCP kubernetes 10.0.0.1 <none> 443/TCP mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP prometheus 10.0.0.112 <nodes> 9090:30003/TCP
- 95. ➤ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) grafana 10.0.0.120 <nodes> 3000:30002/TCP internal-prometheus 10.0.0.39 <none> 9090/TCP kubernetes 10.0.0.1 <none> 443/TCP mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP prometheus 10.0.0.112 <nodes> 9090:30003/TCP Prometheus internal IP
- 96. ➤ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) grafana 10.0.0.120 <nodes> 3000:30002/TCP internal-prometheus 10.0.0.39 <none> 9090/TCP kubernetes 10.0.0.1 <none> 443/TCP mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP prometheus 10.0.0.112 <nodes> 9090:30003/TCP Prometheus external port
- 97. ➤ open http://$(minikube ip):30003
- 98. ➤ open http://$(minikube ip):30003
- 99. ➤ open http://$(minikube ip):30002
- 100. ➤ open http://$(minikube ip):30002
- 104. PromQL Query
- 107. Name your cluster
- 108. Pick your platform
- 110. Run Command
- 111. ➤ kubectl apply -n kube-system -f “<some_url>&t=<some_token>” serviceaccount "weave-flux" created clusterrole "weave-flux" created clusterrolebinding "weave-flux" created secret "flux-git-deploy" created deployment "weave-flux-memcached" created service "weave-flux-memcached" created deployment "weave-flux-agent" created serviceaccount "weave-scope" created clusterrole "weave-scope" created clusterrolebinding "weave-scope" created daemonset "weave-scope-agent" created serviceaccount "weave-cortex" created clusterrole "weave-cortex" created clusterrolebinding "weave-cortex" created deployment "weave-cortex-agent" created service "weave-cortex-agent" created daemonset "weave-cortex-node-exporter" created configmap "weave-cortex-agent-config" created
- 112. ➤ kubectl get pods -n kube-system NAME READY STATUS RESTARTS kube-addon-manager-minikube 1/1 Running 1 kube-dns-910330662-bv35c 3/3 Running 3 kubernetes-dashboard-zj028 1/1 Running 1 weave-cortex-agent-815474457-5q0rg 1/1 Running 0 weave-cortex-node-exporter-5tf88 1/1 Running 0 weave-flux-agent-1731903026-d0gw8 1/1 Running 0 weave-flux-memcached-2601059440-f31vp 1/1 Running 0 weave-scope-agent-6fq0b 1/1 Running 0
- 113. Go to monitoring
- 116. Adding the Prometheus Agent to our app
- 117. ➤ npm install —save epimetheus
- 118. ➤ npm install —save epimetheus Client libraries in: Go, Java, Python, Ruby, Bash, C++, Common Lisp, Elixir, Erlang, Haskell, Lua, .NET, PHP, Rust…
- 120. Very straight forward in most languages
- 121. … Omitted for brevity: Pushing new image to registry Creating new manifest …
- 122. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v2
- 123. ➤ kubectl apply -f https://tinyurl.com/kube-prom-demo-v2 deployment "mighty-fine-fe" configured service "mighty-fine-fe" configured
- 124. ➤ open http://$(minikube ip):30001/metrics
- 125. ➤ open http://$(minikube ip):30001/metrics
- 126. Getting Prometheus to scrape our app
- 128. ➤ kubctl apply -f https://tinyurl.com/kube-prom-monitoring-v2
- 129. ➤ kubctl apply -f https://tinyurl.com/kube-prom-monitoring-v2 deployment "prometheus" configured service "prometheus" configured service "internal-prometheus" configured deployment "grafana" configured service "grafana" configured configmap "prometheus-configmap" configured
- 130. ➤ curl -X POST http://$(minikube ip):30001/-/reload Tell Prometheus to reload its config
- 131. NodeJS Metrics
- 135. … Omitted for brevity: Pushing new image to registry Creating new manifest …
- 136. ➤ kubctl apply -f https://tinyurl.com/kube-prom-monitoring-v3 deployment "prometheus" configured service "prometheus" configured service "internal-prometheus" configured deployment "grafana" configured service "grafana" configured configmap "prometheus-configmap" configured
- 137. ➤ open http://$(minikube ip):30001
- 138. ➤ open http://$(minikube ip):30001
- 139. ➤ open http://$(minikube ip):30001
- 140. Custom Metric
- 142. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5
- 143. Joel York’s SaaS Metrics http://chaotic-flow.com
- 144. Worked Example: Churn rate C × Δt Churn Ratemonth = ΔCcancel
- 145. Worked Example: Churn rate C × Δt Number of cancellations In interval Churn Ratemonth = ΔCcancel
- 146. Worked Example: Churn rate C × Δt Number of cancellations In interval Churn Ratemonth = ΔCcancel Number of customers (at start of interval)
- 147. Worked Example: Churn rate Time interval Number of customers (at start of interval) Number of cancellations In interval Churn Ratemonth = ΔCcancel C × Δt
- 148. Worked Example: Churn rate Churn Ratemonth = ΔCcancel C × Δt
- 149. Worked Example: Churn rate Assumed metrics: total_signups (counter) total_cancels (counter) Churn Ratemonth = ΔCcancel C × Δt
- 150. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Churn Ratemonth = ΔCcancel C × Δt
- 151. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Churn Ratemonth = ΔCcancel C × Δt
- 152. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Churn Ratemonth = ΔCcancel C × Δt t0, t1, t2, t3, t4, t5, t6
- 153. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Data window Churn Ratemonth = ΔCcancel C × Δt
- 154. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Data window Data range (vector) Churn Ratemonth = ΔCcancel C × Δt
- 155. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Data window Data range (vector) Churn Ratemonth = ΔCcancel C × Δt t0, t1, t2, t3, t4, t5, t6 0 2 4 7 9 11 … 0 2 4 7 9 11 … 0 2 4 7 9 11 … 0 2 4 7 9 11 … 0 2 4 7 9 11 … 0 2 4 7 9 11 … 0 2 4 7 9 11 …
- 156. Worked Example: Churn rate ΔCcancel = rate(total_cancels[1m]) Base metric (scalar) Data window Data range (vector) Built-in rate function Churn Ratemonth = ΔCcancel C × Δt
- 157. Worked Example: Churn rate C = (total_signups offset 1m) - (total_cancels offset 1m) Churn Ratemonth = ΔCcancel C × Δt
- 158. Worked Example: Churn rate C = (total_signups offset 1m) - (total_cancels offset 1m) Churn Ratemonth = ΔCcancel C × Δt
- 159. Worked Example: Churn rate C = (total_signups offset 1m) - (total_cancels offset 1m) One month ago Churn Ratemonth = ΔCcancel C × Δt
- 160. Worked Example: Churn rate Churn Ratemonth = rate(total_cancels[1m]) / ((total_signups offset 1m) - (total_cancels offset 1m)) Churn Ratemonth = ΔCcancel C × Δt
- 161. Worked Example: Churn rate Churn Ratemonth = rate(total_cancels[1m]) / ((total_signups offset 1m) - (total_cancels offset 1m)) Churn Ratemonth = ΔCcancel C × Δt
- 162. Getting started with Kubernetes1 2 3 4 The monitoring maturity ladder Whitebox vs blackbox monitoring Monitoring with Prometheus Using PromQL5 Review
- 163. References & useful links - https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html - http://www.ncsysadmin.org/meetings/1010/Monitoring_and_Alerting.pdf - https://www.oreilly.com/ideas/monitoring-distributed-systems - https://www.slideshare.net/brianbrazil/monitoring-what-matters-the-prometheus-approach-to- whitebox-monitoring-berlin-ops-summit-2016 Thank You! Brice Fernandes @fractallambda @weaveworks Slides: https://tinyurl.com/prometheus-kubernetes-slides Code: https://tinyurl.com/prometheus-kubernetes-code Video: https://tinyurl.com/cloud-native-2017 https://weave.works