Creating the PromQL Transpiler for Flux by Julius Volz, Co-Founder | Prometheus
1 like1,782 views
The document discusses the creation of a PromQL transpiler for Flux by Julius Volz, aimed at integrating Prometheus with InfluxDB and making Grafana dashboards reusable. It outlines the challenges of mapping data models between PromQL and Flux, as well as detailing the necessary implementation steps and features to support various query types. Additionally, it emphasizes the experimental status of the transpiler and existing code forks aimed at enhancing functionality and performance.
![Data Model: Prometheus
Series identified once, then multiple timestamp / value pairs:
<identifier> → [ (t0, v0), (t1, v1), … ]
What is this? int64 (ms) float64
Data Model: Prometheus
Series identified once, then multiple timestamp / value pairs:
<identifier> → [ (t0, v0), (t1, v1), … ]
What is this? int64 (ms) float64](https://image.slidesharecdn.com/creatingthepromqltranspilerforfluxbyjuliusvolz-190617135832/85/Creating-the-PromQL-Transpiler-for-Flux-by-Julius-Volz-Co-Founder-Prometheus-11-320.jpg)
![Real-World Transpilation Result
PromQL Flux
http_requests_total
Start: 1550768830.123
End: 1550767900.321
Step: 150
from(bucket: "prometheus")
|> range(start: 2019-02-21T16:45:30.123Z, stop: 2019-02-21T16:51:40.321Z)
|> filter(fn: (r) =>
(r._field == "http_requests_total" and r._measurement == "prometheus"))
|> window(every: 150000000000ns, period: 5m, offset: 130123000000ns)
|> filter(fn: (r) =>
(r._stop >= 2019-02-21T16:50:30.123Z and r._start <= 2019-02-21T16:46:40.321Z))
|> last()
|> timeShift(duration: 0ns)
|> drop(columns: ["_measurement"])
|> duplicate(column: "_stop", as: "_time")
Real-World Transpilation Result
PromQL Flux
http_requests_total
Start: 1550768830.123
End: 1550767900.321
Step: 150
from(bucket: "prometheus")
|> range(start: 2019-02-21T16:45:30.123Z, stop: 2019-02-21T16:51:40.321Z)
|> filter(fn: (r) =>
(r._field == "http_requests_total" and r._measurement == "prometheus"))
|> window(every: 150000000000ns, period: 5m, offset: 130123000000ns)
|> filter(fn: (r) =>
(r._stop >= 2019-02-21T16:50:30.123Z and r._start <= 2019-02-21T16:46:40.321Z))
|> last()
|> timeShift(duration: 0ns)
|> drop(columns: ["_measurement"])
|> duplicate(column: "_stop", as: "_time")](https://image.slidesharecdn.com/creatingthepromqltranspilerforfluxbyjuliusvolz-190617135832/85/Creating-the-PromQL-Transpiler-for-Flux-by-Julius-Volz-Co-Founder-Prometheus-38-320.jpg)
![Complex Expressions
How to transpile complex expressions like:
sum by(path) (rate(http_requests_total{status="500"}[1m]))
/
sum by(path) (rate(http_requests_total[1m]))
* 100 > 5
Complex Expressions
How to transpile complex expressions like:
sum by(path) (rate(http_requests_total{status="500"}[1m]))
/
sum by(path) (rate(http_requests_total[1m]))
* 100 > 5](https://image.slidesharecdn.com/creatingthepromqltranspilerforfluxbyjuliusvolz-190617135832/85/Creating-the-PromQL-Transpiler-for-Flux-by-Julius-Volz-Co-Founder-Prometheus-40-320.jpg)
![Complex Expressions
🠆 Start at root node, transpile all child nodes recursively
🠆 Use transpiled Flux children in parent Flux node
>
5*
100/
sum by(path)
rate rate
http_requests_total{status="500"}[1m] http_requests_total[1m]
sum by(path)
Complex Expressions
🠆 Start at root node, transpile all child nodes recursively
🠆 Use transpiled Flux children in parent Flux node
>
5*
100/
sum by(path)
rate rate
http_requests_total{status="500"}[1m] http_requests_total[1m]
sum by(path)](https://image.slidesharecdn.com/creatingthepromqltranspilerforfluxbyjuliusvolz-190617135832/85/Creating-the-PromQL-Transpiler-for-Flux-by-Julius-Volz-Co-Founder-Prometheus-41-320.jpg)
Creating the PromQL Transpiler for Flux by Julius Volz, Co-Founder | Prometheus
- 1. Julius Volz - Prometheus Co-Founder Creating the PromQL Transpiler for Flux
- 2. Overview
- 3. InfluxDB as long-term storage for Prometheus Reuse Prometheus Grafana dashboards with InfluxDB? 🠆 Support PromQL in InfluxDB via Flux! Build a transpiler. Prometheus Motivation Grafana PromQL query write InfluxDB as long-term storage for Prometheus Reuse Prometheus Grafana dashboards with InfluxDB? 🠆 Support PromQL in InfluxDB via Flux! Build a transpiler. Prometheus Motivation Grafana PromQL query write
- 4. What Needs to be Done? Simple spec on the surface: ...such that results are equivalent (what does this mean?) AST = Abstract Syntax Tree Transpiler PromQL AST Flux AST Input Output What Needs to be Done? Simple spec on the surface: ...such that results are equivalent (what does this mean?) AST = Abstract Syntax Tree Transpiler PromQL AST Flux AST Input Output
- 5. Hidden Questions How to map the data models? Is every feature translatable? If not, can we approximate it? Hidden Questions How to map the data models? Is every feature translatable? If not, can we approximate it?
- 6. Additionally... Make it usable: ● Add PromQL HTTP API to InfluxDB ● Allow data ingestion in expected format Additionally... Make it usable: ● Add PromQL HTTP API to InfluxDB ● Allow data ingestion in expected format
- 7. Integrating Everything Prometheus Grafana PromQL API Transpiler Flux EngineResult Converter PromQL AST Flux AST Flux result PromQL result DBData Mapper Series data Query Store data PromQL Query Integrating Everything Prometheus Grafana PromQL API Transpiler Flux EngineResult Converter PromQL AST Flux AST Flux result PromQL result DBData Mapper Series data Query Store data PromQL Query
- 8. Challenges ● Need to know both Flux and PromQL ● Early days for Flux ● Different data and query models Challenges ● Need to know both Flux and PromQL ● Early days for Flux ● Different data and query models
- 9. Data Models
- 10. Data Model: Prometheus Only metrics / time series No logs / events Data Model: Prometheus Only metrics / time series No logs / events
- 11. Data Model: Prometheus Series identified once, then multiple timestamp / value pairs: <identifier> → [ (t0, v0), (t1, v1), … ] What is this? int64 (ms) float64 Data Model: Prometheus Series identified once, then multiple timestamp / value pairs: <identifier> → [ (t0, v0), (t1, v1), … ] What is this? int64 (ms) float64
- 12. Data Model: Prometheus How are series identified? http_requests_total{job="nginx",instance="1.2.3.4:80",path="/home"} metric name labels Data Model: Prometheus How are series identified? http_requests_total{job="nginx",instance="1.2.3.4:80",path="/home"} metric name labels
- 13. Data Model: Flux For both events and metrics Data Model: Flux For both events and metrics
- 14. Data Model: Flux (Briefly) Stream: set of tables Tables: typed columns, grouping key, and records Columns: 8 different data types (+ values can also be null) Special columns: _time, _start, _stop, _measurement, _field Data Model: Flux (Briefly) Stream: set of tables Tables: typed columns, grouping key, and records Columns: 8 different data types (+ values can also be null) Special columns: _time, _start, _stop, _measurement, _field
- 15. Data Model Mapping ● Metric name 🠆 _field ● _measurement = "prometheus" (see Telegraf #4415) ● Label name 🠆 column name (exception: _foo 🠆 ~_foo) ● Label value 🠆 column value ● Sample value 🠆 _value ● Sample timestamp 🠆 _time Data Model Mapping ● Metric name 🠆 _field ● _measurement = "prometheus" (see Telegraf #4415) ● Label name 🠆 column name (exception: _foo 🠆 ~_foo) ● Label value 🠆 column value ● Sample value 🠆 _value ● Sample timestamp 🠆 _time
- 16. Data Model Mapping Example http_requests_total{method="POST"} 🠆 1043.0 @ 1550767776.382 🠆 1070.0 @ 1550767786.344 🠆 ... Column name _measurement method _field _start _stop _time _value Grouping key? true true true true true false false Column type string string string time time time float "prometheus" "POST" "http_requests_total" 2019-02-21T16:49:00Z 2019-02-21T16:50:00Z 2019-02-21T16:49:36.382Z 1043.0 "prometheus" "POST" "http_requests_total" 2019-02-21T16:49:00Z 2019-02-21T16:50:00Z 2019-02-21T16:49:46.382Z 1070.0 ... ... ... ... ... ... ... Data Model Mapping Example http_requests_total{method="POST"} 🠆 1043.0 @ 1550767776.382 🠆 1070.0 @ 1550767786.344 🠆 ... Column name _measurement method _field _start _stop _time _value Grouping key? true true true true true false false Column type string string string time time time float "prometheus" "POST" "http_requests_total" 2019-02-21T16:49:00Z 2019-02-21T16:50:00Z 2019-02-21T16:49:36.382Z 1043.0 "prometheus" "POST" "http_requests_total" 2019-02-21T16:49:00Z 2019-02-21T16:50:00Z 2019-02-21T16:49:46.382Z 1070.0 ... ... ... ... ... ... ...
- 17. Query Models
- 18. PromQL Query Types Instant Queries + Range Queries PromQL Query Types Instant Queries + Range Queries
- 19. PromQL Instant Queries Evaluate PromQL expression at single timestamp: PromQL Instant Queries Evaluate PromQL expression at single timestamp:
- 20. PromQL Instant Queries Input: ● PromQL Expression ● Evaluation timestamp Output: List of result time series: ● Single sample for each series (for vector selectors) ● Output timestamp is evaluation timestamp PromQL Instant Queries Input: ● PromQL Expression ● Evaluation timestamp Output: List of result time series: ● Single sample for each series (for vector selectors) ● Output timestamp is evaluation timestamp
- 21. Instant Query Example Simplest Example: Series selection Instant Query Example Simplest Example: Series selection
- 22. Raw DataRaw Data
- 23. Instant Query Example: http_requests_totalInstant Query Example: http_requests_total
- 24. Instant Query Example: http_requests_total (Result)Instant Query Example: http_requests_total (Result)
- 25. Instant Query Example: http_requests_totalInstant Query Example: http_requests_total
- 26. Instant Query Example: http_requests_total (Empty Result)Instant Query Example: http_requests_total (Empty Result)
- 27. PromQL Range Queries Evaluate PromQL expression at several timestamps along a range: PromQL Range Queries Evaluate PromQL expression at several timestamps along a range:
- 28. PromQL Range Queries Semantically: Many subsequent instant queries Under the hood: Optimized PromQL Range Queries Semantically: Many subsequent instant queries Under the hood: Optimized
- 29. PromQL Range Queries Input: ● PromQL Expression ● Time range (start, end) ● Step (output resolution) Output: List of result time series: ● Multiple samples for each series PromQL Range Queries Input: ● PromQL Expression ● Time range (start, end) ● Step (output resolution) Output: List of result time series: ● Multiple samples for each series
- 30. Raw DataRaw Data
- 31. Range Query Example: http_requests_totalRange Query Example: http_requests_total
- 32. Range Query Example: http_requests_total (Results)Range Query Example: http_requests_total (Results)
- 33. Flux Query Model (Briefly) Input: ● Flux script: contains both data source (bucket) and all time parameters ● Usually: from() |> range() |> filter() |> window() |> ... Output: ● Stream of arbitrary Flux tables Flux Query Model (Briefly) Input: ● Flux script: contains both data source (bucket) and all time parameters ● Usually: from() |> range() |> filter() |> window() |> ... Output: ● Stream of arbitrary Flux tables
- 35. PromQL Range Query: http_requests_totalPromQL Range Query: http_requests_total
- 36. Flux Transpilation Draft from("prometheus") |> range(...) 🠆 range boundaries? |> filter(...) // Filter series identity here. |> window(...) 🠆 windowing parameters? |> <...more post-processing...> Flux Transpilation Draft from("prometheus") |> range(...) 🠆 range boundaries? |> filter(...) // Filter series identity here. |> window(...) 🠆 windowing parameters? |> <...more post-processing...>
- 37. Flux Range and WindowingFlux Range and Windowing
- 38. Real-World Transpilation Result PromQL Flux http_requests_total Start: 1550768830.123 End: 1550767900.321 Step: 150 from(bucket: "prometheus") |> range(start: 2019-02-21T16:45:30.123Z, stop: 2019-02-21T16:51:40.321Z) |> filter(fn: (r) => (r._field == "http_requests_total" and r._measurement == "prometheus")) |> window(every: 150000000000ns, period: 5m, offset: 130123000000ns) |> filter(fn: (r) => (r._stop >= 2019-02-21T16:50:30.123Z and r._start <= 2019-02-21T16:46:40.321Z)) |> last() |> timeShift(duration: 0ns) |> drop(columns: ["_measurement"]) |> duplicate(column: "_stop", as: "_time") Real-World Transpilation Result PromQL Flux http_requests_total Start: 1550768830.123 End: 1550767900.321 Step: 150 from(bucket: "prometheus") |> range(start: 2019-02-21T16:45:30.123Z, stop: 2019-02-21T16:51:40.321Z) |> filter(fn: (r) => (r._field == "http_requests_total" and r._measurement == "prometheus")) |> window(every: 150000000000ns, period: 5m, offset: 130123000000ns) |> filter(fn: (r) => (r._stop >= 2019-02-21T16:50:30.123Z and r._start <= 2019-02-21T16:46:40.321Z)) |> last() |> timeShift(duration: 0ns) |> drop(columns: ["_measurement"]) |> duplicate(column: "_stop", as: "_time")
- 40. Complex Expressions How to transpile complex expressions like: sum by(path) (rate(http_requests_total{status="500"}[1m])) / sum by(path) (rate(http_requests_total[1m])) * 100 > 5 Complex Expressions How to transpile complex expressions like: sum by(path) (rate(http_requests_total{status="500"}[1m])) / sum by(path) (rate(http_requests_total[1m])) * 100 > 5
- 41. Complex Expressions 🠆 Start at root node, transpile all child nodes recursively 🠆 Use transpiled Flux children in parent Flux node > 5* 100/ sum by(path) rate rate http_requests_total{status="500"}[1m] http_requests_total[1m] sum by(path) Complex Expressions 🠆 Start at root node, transpile all child nodes recursively 🠆 Use transpiled Flux children in parent Flux node > 5* 100/ sum by(path) rate rate http_requests_total{status="500"}[1m] http_requests_total[1m] sum by(path)
- 42. Supporting PromQL Features PromQL: Lots of specific semantics and functionality. Not everything supported yet. ● sum(), avg(), etc. 🠆 Map into existing Flux functionality ● rate(), holt_winters(), etc. 🠆 Build custom Flux helper functions ● Subqueries etc. 🠆 Implement with similar semantics ● Staleness markers, special float values etc. 🠆 Not sure yet ● Non-constant scalar function params etc. 🠆 Won't support? Supporting PromQL Features PromQL: Lots of specific semantics and functionality. Not everything supported yet. ● sum(), avg(), etc. 🠆 Map into existing Flux functionality ● rate(), holt_winters(), etc. 🠆 Build custom Flux helper functions ● Subqueries etc. 🠆 Implement with similar semantics ● Staleness markers, special float values etc. 🠆 Not sure yet ● Non-constant scalar function params etc. 🠆 Won't support?
- 43. Current State Branches in private forks: ● Flux transpiler: https://github.com/juliusv/flux/tree/transpile-promql-flux ● InfluxDB integration: https://github.com/juliusv/influxdb/tree/promflux-api Code still experimental Binary ops require new join features (#1297 or #1219) Speed issues ➞ Determine viability after Flux optimizations Current State Branches in private forks: ● Flux transpiler: https://github.com/juliusv/flux/tree/transpile-promql-flux ● InfluxDB integration: https://github.com/juliusv/influxdb/tree/promflux-api Code still experimental Binary ops require new join features (#1297 or #1219) Speed issues ➞ Determine viability after Flux optimizations
- 44. Demo