Streaming with Structure | Kate Stanley and Salma Saeed, IBM

Streaming with
Structure
Apache Kafka
Kate Stanley
Salma Saeed

Let’s get some coffee…
© 2021 IBM Corporation

Barista Example:
Coffee Shop
Barista
Coffee Lovers
HTTP
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo

Event Driven Architecture
S1
S3
S2
S4
Event-driven messaging backbone
Microservice
publishing
events
Microservice
consuming
events
Event-driven messaging backbone
http://ibm.biz/AdvantagesOfEDA

Barista Example:
Coffee Shop
Coffee Shop
Barista
Barista
Board
Coffee Lovers
Coffee Lovers
HTTP
Orders,
Queue
HTTP
HTTP
https://github.com/cescoffier/quarkus-coffeeshop-demo
Event
Backbone

Data is stored in bytes
Producer Consumer
01 0F 11 01 01 0F 11 01

Data is stored in bytes
Producer Consumer
String Serializer
Integer Serializer
String Deserializer
Integer Deserializer
01 0F 11 01 01 0F 11 01

Serialization
value.serializer="org.apache.kafka.common.serialization.IntegerSerializer”
KafkaProducer<String, String> kafkaProducer =
KafkaProducer.create(vertx, props);
KafkaProducerRecord<String, String> record =
KafkaProducerRecord.create("test", "hello");
Producer Consumer
00 00 00 01
1
2

Serialization
org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class
org.apache.kafka.common.serialization.IntegerSerializer specified in value.serializer
Caused by: java.lang.ClassCastException: java.lang.String incompatible with java.lang.Integer
at org.apache.kafka.common.serialization.IntegerSerializer.serialize(IntegerSerializer.java:19)
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:910)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:870)
Producer Consumer
00 00 00 01
1
2

Deserialization
Producer Consumer
00 00 00 01 00 00 00 01
1

Deserialization
value.deserializer="org.apache.kafka.common.serialization.StringDeserializer”
KafkaConsumer<String, String> kafkaConsumer = KafkaConsumer.create(vertx, props);
Producer Consumer
00 00 00 01
00 00 00 01
1

Deserialization
kafkaConsumer.handler(record -> {
record.value();
record.value().length();
convertToHex(record.value().getBytes(StandardCharsets.UTF_8));
Buffer.buffer(record.value().getBytes(StandardCharsets.UTF_8)).getInt(0)
}
4
“ “
Producer Consumer
00 00 00 01
00 00 00 01
“ ”
1

Deserialization
record.value();
}
00 00 00 01
4
“ “
1
Producer Consumer
1
00 00 00 01
00 00 00 01
1

Deserialization
record.value();
}
00 00 00 02
4
“ “
2
Producer Consumer
00 00 00 02
00 00 00 02
2
2

StringSerializer
Coffee Shop Barista

StringSerializer
Coffee Shop Barista
“id,product,customer,size”

StringSerializer
Coffee Shop Barista
“1234,hot chocolate,Kate,small”

StringSerializer
Coffee Shop Barista
“id,product,marshmallows,customer,size”

StringSerializer
Coffee Shop Barista
“1234,hot chocolate,true,Kate,small”

StringSerializer
Coffee Shop Barista
“1234,hot chocolate,true,Kate,small”
?

JSON gives structure
{
“orderId”: “1234”,
“product”: “hot chocolate”,
“customer”: ”Kate”,
“size”: small
}

JSON gives structure
{
“orderId”: “1234”,
“size”: small,
”marshmallows”: true
}

JSON lacks metadata
{
“orderId”: “1234”,
“size”: small,
”marshmallows”: “yes”
}
?

Adding structure with Schemas

Where to store the schema
Producer Consumer
? ?

Where to store the schema
Producer Consumer

Schema in the message
Producer Consumer
{
“schema”: {
“properties”: {
“orderId”: {“type”: “string”},
“size”: {“type”: ”bool”},
…
},
“orderId”: “1234”,
“size”: small
}

Schema Registry
Producer Consumer
Schema registry

What format to use? Which schema to choose?

Schema Formats
JSON Schema
Google Protocol Buffer
Apache Avro

JSON Schema
A vocabulary that allows you
to annotate and validate JSON documents
Describes your existing data format(s)
Provides clear human- and machine- readable
documentation
Code and schema generators available in Java
https://json-schema.org
{
”type”: “object”,
“properties”: {
“product”: {“type”: “string”},
“customer”: {“type”: “string”},
“size”: {
“type”: “string”,
”enum”: [“small”, “medium”]
}
},
“required”: [“product”, “size”]
}

Protobuf
Mechanism for serializing structured data
Created by Google
Language neutral
Supports generated code in Java, Python, Objective-
C and C++
https://developers.google.com/protocol-buffers/
message Order {
required int32 id = 1;
required string product = 2;
optional bool marshmallows = 3;
}
Order hotChocolate = Order.newBuilder()
.setOrderId(1234)
.setProduct(“hot chocolate”)
.setMarshmallows(true)
.build();
.proto

Apache Avro
A data serialization system that provides:
Rich data structures
A compact, fast, binary data format
Simple integration with dynamic languages
Java, Python and C support
.avsc
{
"type": "record",
"name": "Order",
"fields": [
{"name": "product", "type": "string"},
{"name": "orderId", "type": ”string", "logicalType": "uuid"},
{"name": "marshmallows", "type": "boolean"},
{"name": "size", "type": "enum", "symbols" : ["small", "medium"]}
]
}

Documenting your event-driven
architecture
AsyncAPI
https://www.asyncapi.com

Identifying schemas
id-1 id-2
id-1
v1
v2

Kafka Records
key
value
key
value
key
value
key
value
key
value

Magic bytes
00 00 12 34 af 56 78 9a bc de f0 12 34 56 78 …

Magic bytes
00 00 12 34 af
Magic byte
56 78 9a bc de f0 12 34 56 78 …

Magic bytes
00 00 12 34 af
Integer schema ID
56 78 9a bc de f0 12 34 56 78 …

Magic bytes
00 00 12 34 af 56 78 9a bc de f0 12 34 56 78 …
Serialized data

key:
value:
Magic bytes
00 00 12 34 af 56 78 9a bc de f0 12 34 56 78 …
00 00 12 34 af 56 78 9a bc de f0 12 34 56 78 …

Record Headers
key
value
key
value
key
value
key
value
key
value

Record Headers
key
value
key
value
key
value
key
value
key
value
Header Header Header
Header
Header

Headers
ProducerRecord<String, MySchema> producerRecord =
new ProducerRecord<String, MySchema>(“key”, specificRecord);
producerRecord.headers().add(“value-schema-id”, schema.getIdAsBytes());
producerRecord.headers().add(“value-schema-version”, schema.getVersionAsBytes());
producer.send(producerRecord);

key
value
header
Magic Byte
key
value
header
Record Header

Serializers, Deserializers,
Serdes and Converters

Producer/Consumer
Producer
key.serializer
value.serializer
Consumer
key.deserializer
value.deserializer

Producer/Consumer
Producer
key.serializer
value.serializer
schema.registry.HeaderAvroValueSerializer
Consumer
key.deserializer
value.deserializer
schema.registry.HeaderAvroValueDeserializer

Kafka Connect
Converter
key.converter
value.converter
org.apache.kafka.connect.json.JsonConverter
org.apache.kafka.connect.storage.StringConverter
org.apache.kafka.connect.converters.ByteArrayConverter
schema.registry.HeaderAvroConverter

Kafka Streams
SerDes
key.serde
value.serde
org.apache.kafka.common.serialization.Serdes
Serdes.ByteArray()
Serdes.String()
Serdes.Integer()
schema.registry.HeaderAvroSerdes

Kafka connector Consumer
External System
Kafka Streams
app
Schema registry

Kafka connector Consumer
External System
Kafka Streams
app
Schema registry
Converter
SerDes
Deserializer

Example Schema Registries
Apicurio Registry
Format options: Apache Avro, Google protocol buffers, JSON Schema
Storage options: Kafka, PostgreSQL
API support: Apicurio, IBM, Confluent
License: Apache 2.0
https://github.com/Apicurio/apicurio-registry
Confluent Schema Registry
Format options: Apache Avro, Google protocol buffers, JSON Schema
Storage options: Kafka
API support: Confluent
License: Confluent community license
https://docs.confluent.io/platform/current/schema-registry/index.html#

Summary
Use schemas to get structure
Schema registry provides a centralized store
Choose your favourite format and identification
mechanism

Thank you
Kate Stanley | @katestanley91
Salma Saeed | @salmasaeed07
Getting started with Kafka:
https://developer.ibm.com/components/kafka/
https://kafka.apache.org/quickstart
https://strimzi.io
https://github.com/ibm-messaging/kafka-java-vertx-starter
https://www.apicur.io/registry/docs/apicurio-registry/index.html
https://github.com/Apicurio/apicurio-registry
https://docs.confluent.io/platform/current/schema-registry/index.html#

Streaming with Structure | Kate Stanley and Salma Saeed, IBM

More Related Content

What's hot (20)

Similar to Streaming with Structure | Kate Stanley and Salma Saeed, IBM (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Streaming with Structure | Kate Stanley and Salma Saeed, IBM

Editor's Notes