Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022
0 likes38 views
Alan Pope, an engineering manager at InfluxData, presents various data collection options for InfluxDB, including a new feature for building custom Telegraf binaries. Telegraf serves as an open-source data collection agent with over 300 plugins, allowing users to collect metrics efficiently. The session covers the architecture of InfluxDB, data sources, and different ingestion methods while promoting community resources for support.
![Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022](https://image.slidesharecdn.com/dem03datacollectorsdemoalanpope-221020204223-95e29bc9/85/Alan-Pope-InfluxData-Data-Collectors-InfluxDays-2022-1-320.jpg)
Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022
- 2. Data Collection Demonstration Alan Pope - Engineering Manager, InfluxData
- 3. Discover some of the many data collection options that InfluxDB offers. This session includes a demonstration of a new feature that allows users to quickly build custom Telegraf binaries. Telegraf is InfluxDB’s open source data collection agent that has 300+ plugins. Alan Pope Engineering Manager, InfluxData Alan Pope is an Engineering Manager for Telegraf at InfluxData. He’s been an active Open Source contributor for over a decade. Alan previously worked on Ubuntu and in a past life taught SAP training course. In his spare time, he presents a long-running podcast with friends. He enjoys conversations about the intersection of technology and humanity. Alan loves cats. Data Collectors Demonstration
- 4. Agenda 1. Review Data Collection Options 2. Telegraf Custom Build Feature 3. Demo time
- 5. InfluxDB Platform - Reference Architecture Data Sources Application Workflows Infrastructure Insights Telegraf Client Libraries HTTP Syslog Kubernetes Apache Kafka Python Arduino Node.js JavaScript Go Data Systems Mobile apps Web apps Cloud Services Devices Sensors Databases Networks Message Queues APIs IoT Platforms CRMs InfluxDB Platform IoT Actions InfluxDB Purpose-Built Time Series Database Visualization, Query & Task Engine Collect Downsample Trigger Alert Transform … 200+ Plugins 20+ Languages … New Square Native Ecosystems JMeter NiFi AWS Kinesis Azure Event Hubs GCP PubSub Java .NET/C# PHP Ruby Vector Fluentd
- 6. Data Collection Options Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem • 200+ Telegraf plugins • Regular cadence of releases • Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable • Prometheus scraper (OSS only) • Flux prometheus.from • Flux csv.from(url) • Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device • 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP • Handles batching, chunking, setting right headers, etc. • Why use them? ○ Easy way to get started ○ Need libraries when building custom applications • Source system speaks line protocol • Examples: JMeter, NiFi, Vector, Fluentd • Influx CLI CSV Import • Why use them? ○ Know what you want to monitor, quick and easy integration
- 7. Multiple InfluxDB Ingestion Options Native Collectors Telegraf Client Libraries Scrapers Ecosystem InfluxDB API Interfaces directly with Cloud data sources (in this case, MQTT Brokers) Offers over 300 plugins to collect data from many data sources Available for many popular development languages: Make API Calls to write data to InfluxDB Cloud Scrape Prometheus-formatted metrics from an HTTP-accessible endpoint and store them in InfluxDB. Configure third-party technologies to send line protocol directly to InfluxDB. Build your own with the InfluxDB API, which provides a programmatic interface for interactions with InfluxDB
- 8. Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InfluxDB • Optimized for streaming data Telegraf HTTP Syslog Kubernetes Apache Kafka InfluxDB Purpose-Built Time Series Database Visualization, Query & Task Engine Collect Downsample Trigger Alert Transform 200+ Plugins AWS Kinesis Azure Event Hubs GCP PubSub
- 9. Telegraf Binary growth +18Mb AWS EC2 Metadata processor plugin +10Mb Aliyuncms input plugin +8Mb SQL input plugin
- 10. Telegraf Custom Builder make build_tools ./tools/custom_builder/custom_builder --config ~/telegraf.conf Enable users to easily build custom Telegraf agents only with desired plugins
- 11. How does the custom build feature work?
- 12. Demo: Using Telegraf Custom Build
- 13. Additional Resources Free InfluxDB: OSS or Cloud - influxdata.com/cloud Forums: community.influxdata.com Slack: influxcommunity.slack.com Reddit: r/InfluxData Influx Community (GH): github.com/InfluxCommunity Book: awesome.influxdata.com Docs: docs.influxdata.com Blogs: influxdata.com/blog InfluxDB University: influxdata.com/university How-to guides: docs.influxdata.com/resources/how-to-guides/
- 14. T H A N K Y O U