Samantha Wang [InfluxData] | Data Collection Overview | InfluxDays 2022
0 likes57 views
This document summarizes Samantha Wang's presentation on InfluxDB data collection options. She discussed three main options: native data collection, client libraries, and Telegraf. Native collection allows ingesting data directly from sources without transformation. Client libraries are available for many languages. Telegraf has over 300 plugins and supports ingesting from many sources. Future plans for Telegraf include reducing its binary size, improving the CLI, and adding new plugins.
![Samantha Wang [InfluxData] | Data Collection Overview | InfluxDays 2022](https://image.slidesharecdn.com/dem02datacollectorssamanthawang-221020204002-f3f044a7/85/Samantha-Wang-InfluxData-Data-Collection-Overview-InfluxDays-2022-1-320.jpg)
![Adds [[secretstores.xyz]] section
to Telegraf config.
Can be managed by JOSE
(Javascript Object Signing and
Encryption) based secret store or
your native OS keyring.
Secret Store in Telegraf
[[secretstores.os]]
## Unique identifier for the secret-store.
## This id can later be used in plugins to
reference the secrets
## in this secret-store via
@{<id>:<secret_key>} (mandatory)
id = "secretstore"
## Keyring of the secrets
## In Windows, keys follow a fixed pattern in
the form `<keyring>:<collection>:<key>`. Please
keep this in mind
## when creating secrets with the Windows
credential tool.
# keyring = "telegraf"
# collection = ""
## Allow dynamic secrets that are updated
during runtime of telegraf
# dynamic = false](https://image.slidesharecdn.com/dem02datacollectorssamanthawang-221020204002-f3f044a7/85/Samantha-Wang-InfluxData-Data-Collection-Overview-InfluxDays-2022-25-320.jpg)
Samantha Wang [InfluxData] | Data Collection Overview | InfluxDays 2022
- 2. InfluxDB Data Collectors Samantha Wang - Product Manager, InfluxData
- 3. Connect Learn Build Hear from and meet developers from the InfluxDB Community Be inspired by use cases from our partners and InfluxDB engineers Learn best practices that will help you build great experiences for your projects
- 4. InfluxDB provides multiple methods for ingesting data. Discover the newest options for data collection and what’s in store for the future of Telegraf. Samantha Wang Product Manager, InfluxData Samantha Wang is currently a Product Manager at InfluxData, creator of InfluxDB, the leading time series platform. Samantha is currently responsible for Telegraf, InfluxData’s open source data collection agent. Previous to InfluxData, she worked in product and engineering roles at Gracenote (a Nielsen Company) and Verizon. Samantha holds a degree in Industrial & Systems Engineering from the University of Southern California. InfluxDB Data Collectors
- 5. Agenda 1. Native Data Collection 2. Client Libraries 3. Telegraf
- 6. Multiple InfluxDB Ingestion Options Native Collectors Telegraf Client Libraries Scrapers Ecosystem InfluxDB API Interfaces directly with Cloud data sources (in this case, MQTT Brokers) Offers over 300 plugins to collect data from many data sources Available for many popular development languages: Make API Calls to write data to InfluxDB Cloud Scrape Prometheus-formatted metrics from an HTTP-accessible endpoint and store them in InfluxDB. Configure third-party technologies to send line protocol directly to InfluxDB. Build your own with the InfluxDB API, which provides a programmatic interface for interactions with InfluxDB
- 7. What is Native Data Collection?
- 8. The ability to directly ingest data from popular data sources without the need to do any transformation to Line Protocol or run a Telegraf agent 8 Native Collection
- 9. 9 MQTT Topic Subscriptions • Subscribe to topics being published by their own devices and sent through a message broker directly to the InfluxDB Cloud service • No code or installing any additional software required
- 10. MQTT Ingestion: Telegraf vs Native MQTT
- 11. Simple 3 Step Configuration Define Parsing Rules • Configure Rules for finding timestamps, measurements, tags, and field information in the MQTT messages Configure Broker • Broker URL • Broker TCP/IP Port Number • Broker Authentication Type • Broker Authentication Parameters Configure Topic • Topic Name • InfluxDB Bucket
- 12. What about the Client Libraries?
- 13. Client Libraries
- 14. Improving the user experience • Release updates to client library last Thursday of the month • Added an error message if pivot() is not included • Improved working with timestamps in the pandas dataframe
- 15. What’s new with Telegraf?
- 16. T H A N K Y O U PROJECT NUMBERS
- 17. 300+ Plugins and Telegraf Capabilities Recent improvements: • Apple M1 and RISC-V support • Precisely schedule collection with `collection_offset` • Faster line protocol parsing • Improved MQTT topic and payload parsing • SNMP plugin can parse MIB files directly • Test artifacts on pull requests
- 18. Telegraf Binary growth +18Mb AWS EC2 Metadata processor plugin +10Mb Aliyuncms input plugin +8Mb SQL input plugin
- 19. Telegraf Maintainers Paweł Żak | @zak-pawel Software Engineer, Intel Corporation Giovanni Luisotto | @Trovalo DBA & BI Dev, Quantumdatis Thomas Casteleyn | @Hipska Software Engineer, Super-Visions
- 20. Telegraf Maintainers Paweł Żak | @zak-pawel Software Engineer, Intel Corporation Giovanni Luisotto | @Trovalo DBA & BI Dev, Quantumdatis Thomas Casteleyn | @Hipska Software Engineer, Super-Visions
- 21. What’s next for Telegraf?
- 22. Focus for Telegraf • Empower users to create a customized Telegraf • Reducing the Telegraf binary • Improve the CLI user experience to be logical and consistent • Align with the future of InfluxDB - logging, tracing, and high cardinality
- 23. Telegraf Binary growth +18Mb AWS EC2 Metadata processor plugin +10Mb Aliyuncms input plugin +8Mb SQL input plugin
- 24. Telegraf Custom Builder make build_tools ./tools/custom_builder/custom_builde r --config ~/telegraf.conf Enable users to easily build custom Telegraf agents only with desired plugins
- 25. Adds [[secretstores.xyz]] section to Telegraf config. Can be managed by JOSE (Javascript Object Signing and Encryption) based secret store or your native OS keyring. Secret Store in Telegraf [[secretstores.os]] ## Unique identifier for the secret-store. ## This id can later be used in plugins to reference the secrets ## in this secret-store via @{<id>:<secret_key>} (mandatory) id = "secretstore" ## Keyring of the secrets ## In Windows, keys follow a fixed pattern in the form `<keyring>:<collection>:<key>`. Please keep this in mind ## when creating secrets with the Windows credential tool. # keyring = "telegraf" # collection = "" ## Allow dynamic secrets that are updated during runtime of telegraf # dynamic = false
- 26. CLI Rewrite • Meet developers where they are • Implement CLI best practices and consistencies telegraf <subcommand> <options>
- 27. Continue Releasing Plugins & Features • OPC-UA events based input plugin • CSV Serializer • Azure Monitor Input Plugin • PostgreSQL Output Plugin • Plugin state persistence
- 28. Continue Releasing Plugins & Features • OPC-UA events based input plugin • CSV Serializer • Azure Monitor Input Plugin • PostgreSQL Output Plugin • Plugin state persistence
- 29. Future Breaking Changes in next major release • Plugins and settings deprecation • Processors and Aggregator behavior
- 30. Changes in release schedule Current Future Feature release (2.x) End of every quarter Every month Maintenance release (2.x.x) Every 3 weeks Only if urgent fixes
- 31. InfluxData Community Resources GitHub: https://github.com/influxdata Slack: http://influxdata.com/slack Community Forum: https://community.influxdata.com InfluxDB University: https://university.influxdata.com
- 32. T H A N K Y O U
- 33. Additional Resources Free InfluxDB: OSS or Cloud - influxdata.com/cloud Forums: community.influxdata.com Slack: influxcommunity.slack.com Reddit: r/InfluxData Influx Community (GH): github.com/InfluxCommunity Book: awesome.influxdata.com Docs: docs.influxdata.com Blogs: influxdata.com/blog InfluxDB University: influxdata.com/university How-to guides: docs.influxdata.com/resources/how-to-guides/