![Overflow management
// Set the max size to 10 and fail if reached
@OnOverflow(value = OnOverflow.Strategy.BUFFER, bufferSize = 10)
@Inject @Channel("channel") Emitter<String> emitterWithBuffer;
// [DANGER ZONE] no limit
@OnOverflow(OnOverflow.Strategy.UNBOUNDED_BUFFER)
@Inject @Channel("channel") Emitter<String> danger;
// Drop the new messages if the size is reached
@OnOverflow(OnOverflow.Strategy.DROP)
@Inject @Channel("channel") Emitter<String> dropping;
// Drop the previously sent messages if the size is reached
@OnOverflow(OnOverflow.Strategy.LATEST)
@Inject @Channel("channel") Emitter<String> dropOldMessages;
https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.2/emitter/emitter.html#emitter-overflow](https://image.slidesharecdn.com/kafkaclientsandemitters-211018204916/85/Kafka-clients-and-emitters-15-320.jpg)
Kafka clients and emitters
- 1. Edgar Domingues October 2021 Kafka Client and Emitters
- 2. 01 02 03 04 Agenda Apache Kafka Kafka vs. Message Broker Quarkus Extension for Apache Kafka Testing a Kafka application 2
- 3. Apache Kafka Open-source distributed event streaming platform “event streaming is the practice of • capturing data in real-time from event sources (like databases, sensors, mobile devices, cloud services, and software applications) in the form of streams of events; • storing these event streams durably for later retrieval; • manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and • routing the event streams to different destination technologies as needed.” -- https://kafka.apache.org/documentation/ 3
- 6. Partitioner Producer will decide target partition to place any message, depending on: 1. Partition id, if it's specified within the message 2. key % num partitions, if no partition id is mentioned 3. Round robin if neither partition id nor message key are available in message, meaning only value is available 6
- 8. Kafka vs. Message Broker Kafka 8 • Dumb-Broker and Smart-Consumer • Pull • Topics/Partitions/Groups • Connectors • Stronger guarantees of messages ordering • Persisted events Message Broker (e.g. RabbitMQ • Smart-Broker and Dumb-Consumer • Push • Queues/Exchanges/Routing • Superior support when it comes to routing and filtering messages for consumers to use. • Message Time-To-Live and delayed/scheduled messages • Delivery retries and dead-letter exchanges/queues
- 9. 9 Quarkus Extension for Apache Kafka <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId> </dependency>
- 10. 10 Disable kafka health check # Keep kafka health disabled to avoid leaking the address of the kafka instances quarkus.kafka.health.enabled=false
- 11. 11 Receiving messages from kafka @Incoming("accounts") @Blocking public void consumeAccountEvent(final JsonObject json) { // handle event }
- 12. 12 Configuring Receiver #kafka.bootstrap.servers # from environment mp.messaging.incoming.accounts.connector=smallrye-kafka mp.messaging.incoming.accounts.topic=event-splitter.accounts mp.messaging.incoming.accounts.value.deserializer=io.vertx.kafka.client.serialization.JsonObjectDeserializer mp.messaging.incoming.accounts.group.id=wfm-mercury-accounts mp.messaging.incoming.accounts.auto.offset.reset=earliest mp.messaging.incoming.accounts.security.protocol=SASL_SSL mp.messaging.incoming.accounts.sasl.mechanism=SCRAM-SHA-512 mp.messaging.incoming.accounts.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=${kafka.username} password="${kafka.password}"; mp.messaging.incoming.accounts.ssl.truststore.location=${kafka.truststore.location} mp.messaging.incoming.accounts.ssl.truststore.password=${kafka.truststore.password}
- 13. 13 Sending messages to kafka @Inject @Channel("timeseries-data-reporting") @OnOverflow(value = OnOverflow.Strategy.UNBOUNDED_BUFFER) Emitter<TimeSeriesReportEvent> emitter; void publish(final TimeSeriesReportEvent event) { // emitter.send(event); String key = event.getAccountId() + "-" + event.getQueueId(); emitter.send(KafkaRecord.of(key, event)); }
- 14. 14 Configuring Emitter #kafka.bootstrap.servers # from environment mp.messaging.outgoing.timeseries-data-reporting.connector=smallrye-kafka mp.messaging.outgoing.timeseries-data-reporting.topic=wfm-timeseries.timeseries-data-reporting mp.messaging.outgoing.timeseries-data-reporting.value.serializer=io.quarkus.kafka.client.serialization.JsonbSerializer mp.messaging.outgoing.timeseries-data-reporting.security.protocol=SASL_SSL mp.messaging.outgoing.timeseries-data-reporting.sasl.mechanism=SCRAM-SHA-512 mp.messaging.outgoing.timeseries-data-reporting.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=${kafka.username} password="${kafka.password}"; mp.messaging.outgoing.timeseries-data-reporting.ssl.truststore.location=${kafka.truststore.location} mp.messaging.outgoing.timeseries-data-reporting.ssl.truststore.password=${kafka.truststore.password} mp.messaging.outgoing.timeseries-data-reporting.compression.type=zstd
- 15. Overflow management // Set the max size to 10 and fail if reached @OnOverflow(value = OnOverflow.Strategy.BUFFER, bufferSize = 10) @Inject @Channel("channel") Emitter<String> emitterWithBuffer; // [DANGER ZONE] no limit @OnOverflow(OnOverflow.Strategy.UNBOUNDED_BUFFER) @Inject @Channel("channel") Emitter<String> danger; // Drop the new messages if the size is reached @OnOverflow(OnOverflow.Strategy.DROP) @Inject @Channel("channel") Emitter<String> dropping; // Drop the previously sent messages if the size is reached @OnOverflow(OnOverflow.Strategy.LATEST) @Inject @Channel("channel") Emitter<String> dropOldMessages; https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.2/emitter/emitter.html#emitter-overflow
- 16. 16 Testing a Kafka application <dependency> <groupId>io.smallrye.reactive</groupId> <artifactId>smallrye-reactive-messaging-in-memory</artifactId> <scope>test</scope> </dependency>
- 17. 17 Kafka Test Resource Lifecycle Manager public class KafkaTestResourceLifecycleManager implements QuarkusTestResourceLifecycleManager { @Override public Map<String, String> start() { return InMemoryConnector.switchIncomingChannelsToInMemory("accounts"); } @Override public void stop() { InMemoryConnector.clear(); } }
- 18. 18 Test class @QuarkusTest @QuarkusTestResource(KafkaTestResourceLifecycleManager.class) class KafkaListenerIT { @Inject @Any InMemoryConnector connector;
- 19. 19 Test receiving messages from kafka @Test void handleAccountEvent() { // Arrange InMemorySource<JsonObject> emitter = connector.source("accounts"); // Act emitter.send(...); // Assert // assert event handled as expected…
- 20. 20 Test sending messages to Kafka @Test public void sendTimeSeriesReportEvent() { // Arrange InMemorySink<TimeSeriesReportEvent> events = connector.sink("timeseries-data-reporting"); //... // Act service.publish(event); // Assert assertThat(events.received().size(), is(1)); // assert event key and payload as expected... }
- 21. Questions? 21
- 22. 22