InfluxData Webinar 16 June, 2020 - How to Create a Telegraf Parser Plugin for Data Stored in Kafka

Is Your Data in an Exo/c Format Stored in Ka7a?
Let's write a Telegraf Plugin! The Case of Avro.
16 June, 2020
Webinar
Emanuele Falzone, Ph.D. Student @ Politecnico di Milano

Emanuele Falzone
Ph.D Student @ Politecnico di Milano
❤ Free and Open Source Software
Contacts
- emanuele.falzone@polimi.it
- emanuelefalzone.com
- github.com/emanuele-falzone

Outline
- Introdution
- Data Lifeycle
- Use Cases
- Customer Relationship Management System
- Apache Avro
- Telegraf
- Bank transactions
- Conclusions
Disclaimer
This webinar requires a grasp
knowlegde on what are Kafka
and InfluxDB
Emanuele Falzone - InﬂuxData Webinar - 16 June, 2020 3

ingestion storage analysis visualization
Data Lifecycle
Emanuele Falzone - InfluxData Webinar - 16 June, 2020 4

Data Lifecycle
customapplication

Architecture
application
custom
application
custom
without telegraf with telegraf

Data Lifecycle
customapplication

Use Case
Customer Relationship Management System
- Available for different platforms: web, ios, android
- Different club status: blonde, silver, gold, platinum
- Customers continuously provide ratings
Code available at https://github.com/emanuele-falzone/influxdata-webinar-crm-ratings-use-case

Customer Relationship Management

Customer Rela>onship Management
customer data
ratings
RATINGS-WITH-CUSTOMER-DATA
avro
{
"rating_id": 5313,
"user_id": 3,
"stars": 5,
"route_id": 6975,
"rating_time": 1519304105213,
"channel": "web",
"message": "awesome!"
}
{
"id": 3,
"first_name": "Merilyn",
"last_name": "Doughartie",
"email": "mdoughartie1@dedecms.com",
"gender": "Female",
"club_status": "platinium",
"comments": "none"
}
{
"stars": 5,
"rating_time": 1519304105213,
"channel": "web",
"gender": "Female",
"club_status": "platinium"
}

- binary format
- self describing with schema embedded in the data itself
01000101 10110011
int 42
- given the schema you can automaticcally generate encoder/decoder
Apache

Apache
{
"type": "record",
"name": "KsqlDataSourceSchema",
"namespace": "io.confluent.ksql.avro_schemas",
"fields": [
{
"name": ”RATING_TIME",
"type": [
"null",
"long"
],
"default": null
},
{
"name": "STARS",
"type": [
"null",
"int"
],
"default": null
},
. . .
]
}
{
”RATING_TIME": {
"long": 1591557033850
},
"CHANNEL": {
"string": "iOS-test"
},
"STARS": {
"int": 2
},
"CLUB_STATUS": {
"string": "gold"
},
”GENDER": {
"string": ”Female"
}
}
schema
value

{
"type": "record",
"fields": [
{
"name": "TIMESTAMP",
"type": [
"null",
"long"
],
"default": null
},
{
"name": "STARS",
"type": [
"null",
"int"
],
"default": null
},
. . .
]
}
{
"TIMESTAMP": {
"long": 1591557033850
},
"RATING_ID": {
"long": 12
},
"CHANNEL": {
},
"STARS": {
"int": 2
},
"MESSAGE": {
"string": "meh"
},
"CLUB_STATUS": {
"string": "gold"
},
. . .
}
schema
value
encode0xA0 0xB1 0x47 0x06 0x90 . . .
decode
{
"type": "record",
"fields": [
{
"type": [
"null",
"long"
],
"default": null
},
{
"name": "STARS",
"type": [
"null",
"int"
],
"default": null
},
. . .
]
}
{
"TIMESTAMP": {
"long": 1591557033850
},
"RATING_ID": {
"long": 12
},
"CHANNEL": {
},
"STARS": {
"int": 2
},
"MESSAGE": {
"string": "meh"
},
"CLUB_STATUS": {
"string": "gold"
},
. . .
}
schema
value
Apache

Apache messages in Kafka
0x20 0x45 0x37 0x74 0x61 . . . 0xA0 0xB1 0x47 0x06 0x90 . . .
0x20 0x45 0x37 0x74 0x61 . . . 0xA1 0xE1 0x60 0x00 0x00 . . .
0x20 0x45 0x37 0x74 0x61 . . . 0xA3 0xB5 0x00 0x32 0x54 . . .
m bytes
Binary Message
n bytes
Schema

Schema registry
id schema
1
{
"type": "record",
"fields": [
{
"type": [
"null",
"long"
],
"default": null
},
{
"name": "STARS",
"type": [
"null",
"int"
],
"default": null
},
. . .
]
}
GET /schemas/ids/%d
application/json

Kafka + schema registry
0x00 0x00 0x00 0x00 0x01 0xA0 0xB1 0x47 0x06 0x90 . . .
0x00 0x00 0x00 0x00 0x01 0xA1 0xE1 0x60 0x00 0x00 . . .
0x00 0x00 0x00 0x00 0x01 0xA3 0xB5 0x00 0x32 0x54 . . .
4 bytes
Schema ID
m bytes
Binary Message
schema
registry
1 byte
Magic Byte

Architecture
schema
registry
kafka

- the most used tool to feed data to InfluxDB
- written in golang
- Open Source!
- plugin architeture
- more than 200 plugins
- documentation and examples
- easy to deploy
- docker container with config file

Plugin architecture
input
metric
parser
plugins
plugins
telegraf
data flow
telegraf boundary
bytes/string

Telegraf metric
closely based on InfluxDB data model and contain four main components:
- Measurement name: Description and namespace for the metric.
- Tags: Key/Value string pairs and usually used to identify the metric.
- Fields: Key/Value pairs that are typed and usually contain the metric data.
- Timestamp: Date and time associated with the fields.
This metric type exists only in memory and must be converted to a concrete representation in order to be
transmitted or viewed

Plugin architecture
input
metric
parser serializer
TO BE NOTICED: there are also processors and aggregators plugins, but we are not using them.
plugins
output
plugins
pluginsplugins
telegraf
data flow
telegraf boundary

Plugins
input plugins:
file, kafka, influx, mqtt, http, socket, event-hub,
kinesis, cloud-pubsub, mysql, mongodb,
couchdb, postgresql, . . .
parser plugins:
json, csv, form-urlencoded, influx, wavefront, . . .
serializer plugins = input parser
output plugins = input plugins

telegraf.conf
# Input plugin
[[inputs.file]]
files = [”/tmp/sample.csv"]
# Parser plugin
data_format = "csv"
csv_header_row_count = 1
csv_delimiter = ","
csv_measurement_column = "measurement"
csv_timestamp_column = "time"
# Output plugin
[[outputs.influxdb_v2]]
urls = ["http://influxdb:9999"]
token = "d2VsY29tZQ=="
organization = "polimi"
bucket = "bucket”
# Serializer plugin
data_format = ”influx"

Plugin architecture
influxdb-v2
outputkafka
consumer telegraf
metric
avro
parser
influx
serializer
missing!!!
data flow
telegraf boundary

From Apache to Telegraf metric
{
. . .
"fields": [
{
"name": "RATING_TIME",
. . .
},
{
"name": "CHANNEL",
. . .
},
{
"name": "STARS",
. . .
},
{
"name": "CLUB_STATUS",
. . .
},
{
"name": "GENDER",
. . .
}
]
}
TIMESTAMP
TAG
FIELD
TAG
TAG
MEASUREMENT=ratings

telegraf.conf
# Input plugin
[[inputs.kafka_consumer]]
brokers = ["kafka:29092"]
topics = ["RATINGS_WITH_CUSTOMER_DATA"]
# Parser plugin
data_format = "avro"
avro_measurement = "ratings"
avro_tags = ["CHANNEL", "CLUB_STATUS", "GENDER"]
avro_fields = ["STARS"]
avro_timestamp = "RATING_TIME"
avro_timestamp_format = "unix_ms"
avro_schema_registry = "http://schema-registry:8081"
# Output plugin
[[outputs.influxdb_v2]]
urls = ["http://influxdb:9999"]
token = "d2VsY29tZQ=="
organization = "polimi"
bucket = "bucket”
# Serializer plugin
data_format = ”influx"

how can I make telegraf
understand avro messages ?

Code
Gopkg.lock
Gopkg.toml
plugins/parsers/avro/parser.go
plugins/parsers/avro/schema_registry.go
plugins/parsers/avro/parser_test.go
plugins/parsers/registry.go
internal/config/config.go
plugins/parsers/avro/README.md
dependencies
parser logic
tests
parser init
telegraf.conf
documentation
Code available at https://github.com/emanuele-falzone/telegraf/tree/avro/plugins/parsers/avro

Talk is cheap.
Show me the code.
~ Linus Torvalds ~

Open Source Contribution

Can you help John?
John works in a bank and wants to visualize the average amount of transactions over time using
InfluxDB.
Here is a sample transaction: transaction,sender=Alice,receiver=Bob amount=28.00 1591644525345
However, messages are sent to Kafka encrypted with the public key of the bank.
Extend telegraf in order to decrypt every transaction and upload the corresponding
measurement to InfluxDB.
Code available at https://github.com/emanuele-falzone/influxdata-webinar-bank-transactions-use-case

Conclusion
- InfluxDB and telegraf in the Data Lifeycle
- Use Cases
- Customer Relationship Management System
- Apache Avro
- Bank Transactions
- Encryption

Questions?
Contacts
- emanuele.falzone@polimi.it
- emanuelefalzone.com
- github.com/emanuele-falzone

We look forward to bringing together our community of
developers in this new format to learn, interact, and share
tips and use cases.
23-24 June, 2020
Virtual Experience
www.influxdays.com

InfluxData Webinar 16 June, 2020 - How to Create a Telegraf Parser Plugin for Data Stored in Kafka

More Related Content

What's hot (20)

Similar to InfluxData Webinar 16 June, 2020 - How to Create a Telegraf Parser Plugin for Data Stored in Kafka (20)

Recently uploaded (20)

InfluxData Webinar 16 June, 2020 - How to Create a Telegraf Parser Plugin for Data Stored in Kafka