Barbara Nelson [InfluxData] | Build, Ship, Repeat: Milestones for InfluxDB | InfluxDays 2022
0 likes15 views
1. InfluxDays 2022 focused on building what developers need for open source, on-premise, and cloud environments using CI/CD pipelines. 2. InfluxDB provides various options for collecting data including native collectors, Telegraf, client libraries, scrapers, and ecosystem integrations that can ingest real-time and streaming data. 3. Data is stored in InfluxDB's compressed tsdb and can handle large workloads of hundreds of gigabytes ingested per hour and tens of thousands of queries per hour.
![Performance – It gets better and better
Twelve months of query performance for our performance test suite.
from(bucket: v.bucket)
|> range(start: v.timeRangeStart)
|> filter(fn: (r) => r["_measurement"] == "cpu")
|> group(columns: ["_field"])
|> aggregateWindow(every: v.windowPeriod, fn: mean)](https://image.slidesharecdn.com/influxdays2022influxdbisgettingbetter-221020203635-4365e3c3/85/Barbara-Nelson-InfluxData-Build-Ship-Repeat-Milestones-for-InfluxDB-InfluxDays-2022-8-320.jpg)
![A few extra goodies
• Explicit schema
• Flux: http and json
import "array"
import "experimental/json"
import "experimental/http/requests"
headers = ["x-api-key": TOKEN_REDACTED,]
url = "https://yfapi.net/v6/finance/quote"
response = requests.get(url:url, params: ["symbols":["AAPL"]], headers: headers)
array.from(rows: [{headers: display(v: response.headers)}])
jsonData = json.parse(data: response.body)
array.from(rows: jsonData.quoteResponse.result)
|> map(fn: (r) => ({ _time: now(),
_field: "marketPrice",
_measurement: "stockQuote",
_value: r.regularMarketPrice,
symbol: r.symbol,
}))
|> yield(name: "ts")](https://image.slidesharecdn.com/influxdays2022influxdbisgettingbetter-221020203635-4365e3c3/85/Barbara-Nelson-InfluxData-Build-Ship-Repeat-Milestones-for-InfluxDB-InfluxDays-2022-9-320.jpg)
Barbara Nelson [InfluxData] | Build, Ship, Repeat: Milestones for InfluxDB | InfluxDays 2022
- 1. InfluxDays 2022 Build, Ship, Repeat Barbara Nelson, VP of Engineering
- 2. We keep building what developers need, wherever you are: • OSS • On-premise • Cloud Barbara Nelson VP, Engineering For Cloud, we have a CI/CD pipeline: • Develop/Test/Stage/Release within hours • Feature flags for sophisticated control over when features ‘go live’ InfluxDB is getting better
- 3. Collecting Data Native Collectors Telegraf Client Libraries Scrapers Ecosystem InfluxDB API Native Collectors are a no-code way to ingest real-time, streaming, messaging event data to InfluxDB. Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. Language and platform-specific packages that provides a programmatic way to interface with InfluxDB. Scrape Prometheus-formatted metrics from an HTTP-accessible endpoint and store them in InfluxDB. Configure third-party technologies to send line protocol directly to InfluxDB. Build your own with the InfluxDB API, which provides a programmatic interface for interactions with InfluxDB
- 4. Storing the data By the numbers (in our multi-tenanted Cloud environment) A few example customer workloads: • One customer is writing 345GB every hour. • Another customer is executing 80k queries per hour. 268TB Compressed tsdb data 10.9B/s Ingest bytes 11M Points/s ~2000 Queries/s
- 5. Querying the data - from the UI
- 6. Save and Reuse queries
- 7. Language enhancements Flux: polymorphic labels Flux: dynamic types Flux: inner, outer, left and right joins Flux: isType Flux modules
- 8. Performance – It gets better and better Twelve months of query performance for our performance test suite. from(bucket: v.bucket) |> range(start: v.timeRangeStart) |> filter(fn: (r) => r["_measurement"] == "cpu") |> group(columns: ["_field"]) |> aggregateWindow(every: v.windowPeriod, fn: mean)
- 9. A few extra goodies • Explicit schema • Flux: http and json import "array" import "experimental/json" import "experimental/http/requests" headers = ["x-api-key": TOKEN_REDACTED,] url = "https://yfapi.net/v6/finance/quote" response = requests.get(url:url, params: ["symbols":["AAPL"]], headers: headers) array.from(rows: [{headers: display(v: response.headers)}]) jsonData = json.parse(data: response.body) array.from(rows: jsonData.quoteResponse.result) |> map(fn: (r) => ({ _time: now(), _field: "marketPrice", _measurement: "stockQuote", _value: r.regularMarketPrice, symbol: r.symbol, })) |> yield(name: "ts")
- 10. Tying it all together - the edge and the cloud InfluxDB Platform Telegraf Client Libraries Native Ecosystem Data Sources Edge Telegraf Client Libraries Native Ecosystem Data Sources Edge Telegraf Client Libraries Native Ecosystem Data Sources Edge
- 11. T H A N K Y O U