![Extract metrics with LogQL
2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:14:20 +0000] "GET /loki.jpeg HTTP/1.1" 200 106186 "-" "hey/0.0.1" "-"
2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:15:20 +0000] "GET /foo HTTP/1.1" 400 256 "-" "hey/0.0.1" "-"
2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:16:20 +0000] "GET /bar HTTP/1.1" 418 10239 "-" "hey/0.0.1" "-"
sum by(path, status_code) (
max_over_time(
{app="nginx"}
| pattern `<_> "GET <path> HTTP/1.1" <status_code> <value> <_>`
| unwrap value
[10m]
)
)](https://image.slidesharecdn.com/chooseyourownadventuretogetstartedwithgrafanaloki-250519191943-d940f7ed/85/Choose-Your-Own-Adventure-to-Get-Started-with-Grafana-Loki-45-320.jpg)
Choose Your Own Adventure to Get Started with Grafana Loki
- 1. Choose Your Own Adventure to Get Started with Grafana Loki
- 2. ● Staff Developer Advocate @Grafana Labs ● Observability, Cloud Native and Open Source ● OSS community organizer - inclusive tech Imma Valls Staff Developer Advocate https://eyeveebee.dev/
- 3. Agenda 1. Why observability 2. Choose your own adventure! 3. A bit of history 4. Running Loki 5. Sending logs to Loki 6. Querying logs from Loki 7. Takeaways 8. Q&A
- 5. Today’s reality: Disparate systems. Disparate data. Today’s reality: disparate systems,disparate data
- 6. Pillars of observability WHAT is happening? WHY is it happening? WHERE is it happening? HOW to fix it?
- 7. Pillars of observability Exemplars & Data links Service discovery & labels LogQL Metric extraction Derived fields Or Automated logging Labels Metrics from spans & Custom query specification Metrics: Is something happening? Logs: What is happening? Traces: Where is it happening? Profiles: How do I fix it? Trace links and stack traces Resource usage over time
- 8. 2. Choose your own adventure
- 9. Choose your Own Adventure with Grafana Loki!
- 10. You will use Mentimeter to choose … … let’s test this out!
- 12. How familiar are you with Loki? 1. I've never heard of Loki 2. I've heard about it, I never took it out for a spin 3. I am using Loki < 3.0 4. I am using the latest Loki 3.x
- 13. 3. A bit of history
- 14. ● Every important action is logged in a text file ● An explosion of new log data from cloud, microservices and IoT ● Being able to collect and analyse logs is crucial for companies to get value out of logs Why are logs important?
- 15. Our Motivation for Loki Current logging and metrics technologies are hard to operate at scale 1 Expensive to scale: people, hardware footprint and licenses 2 Doesnʼt correlate well outside of the vendorʼs stack. Navigation between logs, metrics, and traces is complicated 3
- 16. “What if we create a system like Prometheus, but for Logs?” Tom Wilkie Grafana Labs CPO March of 2018
- 17. Grafana Loki Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus
- 18. Who did we make Loki for? Effective debugging and troubleshooting of applications DevOps Visualise and alert on services/apps performance metrics SREs Build actionable insights from log data and other supported data sources DataEng
- 19. Why do they like Loki? Lower TCO at scale. Highly scalable. Smart indexing Easy to operate, scale and maintain at large scale Logs as metrics. Analytics, Alerting, Predictions, etc Format agnostic. Accepts all log formats JSON, regex, logfmt) Seamless integration with Prometheus
- 20. 03/18 Project started by Tom and David 12/18 Launched at KubeCon NA 12/18 #1 on Hacker News for ~12hrs! 04/19 KubeCon EU: context, live tailing 06/19 0.1.0 Beta release! 08/19 0.2.0 Stability improvements 10/19 0.4.0 Filter & Metric Queries 11/19 1.0.0 1.5TB/10 billion log lines a day in our Production cluster 01/20 1.3.0 Syslog, Frontend, and more encoding 04/20 1.4.0 Binary operations & Statistics API 05/20 1.5.0 Boltdb-shipper & Binary operations 08/20 1.6.0 10x metrics query performance, Lambda support 10/20 Loki v2.0! 12/23 Loki turns 5! 🌟 04/24 Most recent release: Loki v3.0! https://github.com/grafana/loki goo.gl/5DEVH6 A bit of history 20
- 21. KubeCon San Diego 1.0 LogQL v2, New Index 2.0 1.6 Alerting from Logs 2.9 High scale alerting, New query APIs 2.2 Write Ahead Log SSD deployment Mode, Out of order support 2.4 2.6 Multi-tenant Queries, Deletion support 2.7 Stream sharding 2.8 (another) New Index 3.0 Bloom filters, OTel native support
- 22. KubeCon San Diego 1.0 LogQL v2, New Index 2.0 1.6 Alerting from Logs 2.9 High scale alerting, New query APIs 2.2 Write Ahead Log SSD deployment Mode, Out of order support 2.4 2.6 Multi-tenant Queries, Deletion support 2.7 Stream sharding 2.8 (another) New Index 3.0 Bloom filters, OTel native support 10GB/s 50GB/s 100GB/s 1TB/s
- 23. 4. Running Loki
- 24. Single Binary - Easier to get started - Small deployments (100GB/day) - Good for testing How to deploy Loki Simple Scalable Deployment - Easier to scale and monitor - Scale to TBs/day - Separate R/W Path - Suited for non-K8S env Microservices - Most efficient & complex - Very large installations - Fine tuned scaling & ops - Works best with K8S I D Q FE R I D Q FE R I D Q FE R I D I D D FE
- 25. Loki OSS - Self managed - No out of the box security or multi-tenancy Grafana Enterprise Logs - Self hosted - Supported by Grafana Labs - Security out of the box - Multi-tenancy and admin tools Cloud Logs - Fully managed (SaaS) - Scalable and highly available - Multi-tenant - Free Plan - 50GB of Logs, 14-day retention Loki distributions
- 26. 5. Sending logs to Loki
- 27. Loki accepts it all 10 year old legacy app nginx IIS Cloud-native app Linux Kubernetes OpenTelemetry your choice of collector/agent custom format access logs Windows events logfmt systemd journal service discovery API JSON, protobuf
- 30. Logs collection easy with... Grafana Alloy ● Targets discovery for Kubernetes, Syslog, files and more ● Automatically attach labels to your log lines ● Advanced pipeline mechanism for parsing, transforming and filtering your logs ● Build and expose custom metrics from your logs data
- 31. Grafana Alloy +120 components to collect telemetry data Prometheus and OpenTelemetry compatible Pull configurations from Git, S3, HTTP endpoints Flexible, high performance, vendor-neutral Collect Transform Load
- 32. Demo Time - Ingest
- 33. What is our log format of choice? 1. {JSON} 2. Plain text
- 35. 6. Querying logs from Loki
- 36. “LogQL, Like PromQL for Logs”
- 37. Log Data Index 10TB 200MB Think of it more like a table of contents than an index Metadata Indexing Loki does not index the text of logs. Instead, entries are grouped into streams and indexed with labels.
- 38. Prometheus metric ingestion Metric Name Value Labels key-value pairs Timestamp @1600156214... nginx_cpu_usage {app=”nginx”,instance=”1.1..”} 14.4
- 39. Loki log ingestion 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs indexed unindexed GET /about 1034 Debug “page not found”
- 40. A log stream is a stream of log entries with the same labels 2019-10-13T10:01:02.000Z {app=”nginx”,instance=”1.1.1.1”} GET /about 2019-10-13T10:03:04.000Z {app=”nginx”,instance=”1.1.1.1”} GET / 2019-10-13T10:05:06.000Z {app=”nginx”,instance=”1.1.1.1”} GET /help 2019-10-13T10:01:02.000Z {app=”nginx”,instance=”2.2.2.2”} GET /users/1 2019-10-13T10:03:04.000Z {app=”nginx”,instance=”2.2.2.2”} GET /users/2 Log Streams
- 41. Loki log ingestion 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs GET /about 1034 Debug “page not found” “index” “chunks” AWS S3 Google Cloud Storage Azure Blob Storage
- 42. Querying Loki with LogQL Label matcher Filter expression
- 43. Examples - Stream selectors {container=”nginx”} {container=~”nginx|envoy|caddy|traefik”} {namespace=”prod”, app!=”agent”} Examples - Line filter operators {cluster=”us-central-1”} |= “error” != “timeout” {namespace=”prod” } |~ `(?i)error` {container=”nginx”} |= ip("192.168.4.5/16") Filter expression Given matching log streams, scan and match log entries (unindexed) Log selector Filter log streams by matching labels using an index Selecting log streams with LogQL
- 44. Selecting log streams with LogQL Filter expressions ● |= contains string. ● != does not contain string. ● |~ matches regular expression. ● !~ does not match regular expression. Label matchers ● = contains string. ● != does not contain string. ● =~ matches regular expression. ● !~ does not match regular expression.
- 45. Extract metrics with LogQL 2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:14:20 +0000] "GET /loki.jpeg HTTP/1.1" 200 106186 "-" "hey/0.0.1" "-" 2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:15:20 +0000] "GET /foo HTTP/1.1" 400 256 "-" "hey/0.0.1" "-" 2020-05-14 16:14:20 172.20.0.7 - - [14/May/2020:20:16:20 +0000] "GET /bar HTTP/1.1" 418 10239 "-" "hey/0.0.1" "-" sum by(path, status_code) ( max_over_time( {app="nginx"} | pattern `<_> "GET <path> HTTP/1.1" <status_code> <value> <_>` | unwrap value [10m] ) )
- 46. Why extract metrics with LogQL Can’t add instrumentation to an application Need more granularity than existing metrics expose Distributed systems with complex (asynchronous) flow
- 47. Metrics Format Filter Parse logfmt regexp json pattern |= “192.168.0.1” |~ `(?i)error` foo > 10 app =~ “.*loki.*” time >= 23ms lat > 23 and lon > -57 line_format label_format rate bytes_rate bytes_over_time count_over_time unwrap sum_over_time avg_over_time stddev_over_time stdvar_over_time max_over_time min_over_time quantile_over_time topk
- 48. - Detect patterns in your logs - Explore your logs visually - Easy, simple searching - No code required Logs Drilldown I don’t want to learn LogQL!
- 49. Demo Time - Query
- 50. What is your favorite log exploration tool? 1. I am a fan of LogQL (or promQL) 2. I don’t want to learn LogQL, show me Logs Drilldown
- 52. Labeling is Important What makes a good label? ● traceid ❌ ● userid ❌ ● path ❌ ● status_code ❌ ● date ❌ ● latency ❌ ● cluster ✅ ● namespace ✅ ● job ✅ ● app ✅ ● hostname ✅ ● filename ✅
- 53. High Cardinality Beware! ● Streams are unique combinations of label key value ● Avoid high cardinality labels ● Needle in haystack queries (userid, sessionid…)?
- 54. Query Acceleration powered by Bloom Filters {app=”foo”, cluster=”bar”} |= “fizz-1234-buzz-5678” Support Ticket `app` user with id `fizz-1234-buzz-5678` couldn’t add an item to their shopping cart https://grafana.com/docs/loki/latest/operations/bloom-filters/
- 55. 7. Takeaways 1. Loki offers a flexible, easy to operate and scalable log aggregation solution 2. Send logs with different formats and with different Agents / sources 3. Filter and extract metrics with LogQL or easy-to use UI Logs Drilldown
- 56. 7. Where to go Next 1. Grafana Play - https://play.grafana.org 2. Loki documentation - https://grafana.com/docs/loki 3. LogQL simulator - https://grafana.com/docs/loki/latest/query/analyzer 4. Videos from Jay & Nicole 5. Contact the Loki team - https://grafana.com/docs/loki/latest/community/getting-in-touch/ 6. Grafana Cloud Free Tier - https://grafana.com/products/cloud
- 58. Don't forget to rate the talk!
- 59. Thanks!