Robust Operations of Kafka Streams
3 likes3,587 views
The document provides a technical deep dive into Kafka Streams, covering key concepts like topology, tasks, threads, state stores, and error handling. It includes code examples for building resilient Kafka Streams applications, as well as techniques for managing tasks and handling errors during processing. Additionally, it addresses general problem areas and discusses the importance of failover mechanisms.
![6
View the Topology
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [A, B])
--> KSTREAM-AGGREGATE-0000000001
Processor: KSTREAM-AGGREGATE-0000000001 (stores: [count-store])
--> KTABLE-TOSTREAM-0000000002
<-- KSTREAM-SOURCE-0000000000
Processor: KTABLE-TOSTREAM-0000000002 (stores: [])
--> KSTREAM-SINK-0000000003
<-- KSTREAM-AGGREGATE-0000000001
Sink: KSTREAM-SINK-0000000003 (topic: output-topic)
<-- KTABLE-TOSTREAM-0000000002](https://image.slidesharecdn.com/04billbejeck-181022184210/85/Robust-Operations-of-Kafka-Streams-6-320.jpg)
![33
Processing Exceptions DSL
Predicate<String, ValueContainer> errorPredicate =
(k, v) -> v.exception() != null;
Predicate<String, ValueContainer> successPredicate =
(k,v) -> !errorPredicate.test(k,v);
KStream<String, ValueContainer>[] branched = sourceStream.mapValues(errorProneMapper).
branch(errorPredicate, successPredicate);
int errorIndex = 0;
int successIndex = 1;
branched[errorIndex].to(errorReportTopic);
branched[successIndex].mapValues(ValueContainer::formatValue).to(outputTopic);](https://image.slidesharecdn.com/04billbejeck-181022184210/85/Robust-Operations-of-Kafka-Streams-33-320.jpg)
![34
Processing Exceptions DSL
Predicate<String, ValueContainer> errorPredicate = (k, v) -> v.exception() != null;
Predicate<String, ValueContainer> successPredicate = (k,v) ->!errorPredicate.test(k,v);
KStream<String, String>[] branched = sourceStream
.mapValues(errorProneMapper)
.branch(errorPredicate,
successPredicate);
int errorIndex = 0;
int successIndex = 1;
branched[errorIndex].to(errorReportTopic);
branched[successIndex].mapValues(v -> ValueContainer::formatValue).to(outputTopic);](https://image.slidesharecdn.com/04billbejeck-181022184210/85/Robust-Operations-of-Kafka-Streams-34-320.jpg)
![35
Processing Exceptions DSL
Predicate<String, ValueContainer> errorPredicate = (k, v) -> v.exception() != null;
Predicate<String, ValueContainer> successPredicate = (k,v) ->!errorPredicate.test(k,v);
KStream<String, ValueContainer>[] branched = sourceStream.mapValues(errorProneMapper).
branch(errorPredicate,successPredicate);
int errorIndex = 0;
int successIndex = 1;
branched[errorIndex].to(errorReportTopic);
branched[successIndex].mapValues(ValueContainer::formatValue)
.to(outputTopic);](https://image.slidesharecdn.com/04billbejeck-181022184210/85/Robust-Operations-of-Kafka-Streams-35-320.jpg)
Robust Operations of Kafka Streams
- 1. 1 Robust Kafka Streams Streams Technical Deep Dive and Making your Kafka Streams applications more resilient
- 2. 2 Brief Introduction • Worked at Confluent (Streams Team) 1.5 years • Author Kafka Streams in Action
- 3. 3 What We’ll Cover • Concept of Topology and Sub-Topology • Tasks, threads and state stores • Getting Notifications • Error Handling
- 4. 4 Sample Streams Application ... stream = streamBuilder.stream(Arrays.asList(“A”, “B”)) stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()))
- 5. 5 View the Topology Topology topology = streamBuilder.build() topology.describe().toString()
- 6. 6 View the Topology Topologies: Sub-topology: 0 Source: KSTREAM-SOURCE-0000000000 (topics: [A, B]) --> KSTREAM-AGGREGATE-0000000001 Processor: KSTREAM-AGGREGATE-0000000001 (stores: [count-store]) --> KTABLE-TOSTREAM-0000000002 <-- KSTREAM-SOURCE-0000000000 Processor: KTABLE-TOSTREAM-0000000002 (stores: []) --> KSTREAM-SINK-0000000003 <-- KSTREAM-AGGREGATE-0000000001 Sink: KSTREAM-SINK-0000000003 (topic: output-topic) <-- KTABLE-TOSTREAM-0000000002
- 8. 8 View the Topology stream = streamBuilder.stream(Arrays.asList(“A”, “B”)); stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()));
- 9. 9 View the Topology stream = builder.stream(Arrays.asList(“A”, “B”)); stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()));
- 10. 10 View the Topology stream = builder.stream(Arrays.asList(“A”, “B”)); stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()));
- 11. 11 View the Topology stream = builder.stream(Arrays.asList(“A”, “B”)); stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()));
- 13. 13 Updated Sample Streams Application stream = streamBuilder.stream(“A”); secondStream = streamBuilder.stream("B"); stream.groupByKey() .count(Materialized.as(“count-store”)) .toStream() .to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long())); secondStream.filter((k,v) -> k.equals("Foo")) .mapValues(v -> v.substring(0,3)) .to("filtered-output-topic");
- 14. 14 Updated streams Example Topology
- 15. 15 Kafka Streams Task Assignments First Number: Group ID Second Number: Partition ID
- 21. 21 So what does this mean? • N tasks max of N Threads, • Threads > Tasks; Idle threads • Account for #core(s)
- 22. 22 Tasks and State Stores • Stores are assigned Per task
- 23. Task Migration with State
- 24. 24 Standby Tasks
- 25. 25 Standby Tasks
- 26. 26 Standby Task Deployment Considerations
- 27. 27 Standby Task Pros & Cons
- 28. 28 View Task Assignments for (ThreadMetadata threadMetadata : kafkaStreams.localThreadsMetadata()) { LOG.info("Thread {} active {}", threadMetadata.threadName(), threadMetadata.activeTasks()); LOG.info("Thread {} standby {}", threadMetadata.threadName(), threadMetadata.standbyTasks(); }
- 29. 29 The General Problem Areas • Deserialization • Producing • Processing
- 30. 30 Error Handling - Deserializing props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndFailExceptionHandler.class.getName()); props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndContinueExceptionHandler.class.getName());
- 31. 31 Error Handling - Producing props.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG, DefaultProductionExceptionHandler.class.getName());
- 32. 32 Processing Exceptions DSL ValueMapper<String, ValueContainer> errorProneMapper = v -> { ValueContainer valueContainer = new ValueContainer(); try { valueContainer.addValue(v.substring(0, 5)); } catch (Exception e) { valueContainer.setException(e); } return valueContainer; };
- 33. 33 Processing Exceptions DSL Predicate<String, ValueContainer> errorPredicate = (k, v) -> v.exception() != null; Predicate<String, ValueContainer> successPredicate = (k,v) -> !errorPredicate.test(k,v); KStream<String, ValueContainer>[] branched = sourceStream.mapValues(errorProneMapper). branch(errorPredicate, successPredicate); int errorIndex = 0; int successIndex = 1; branched[errorIndex].to(errorReportTopic); branched[successIndex].mapValues(ValueContainer::formatValue).to(outputTopic);
- 34. 34 Processing Exceptions DSL Predicate<String, ValueContainer> errorPredicate = (k, v) -> v.exception() != null; Predicate<String, ValueContainer> successPredicate = (k,v) ->!errorPredicate.test(k,v); KStream<String, String>[] branched = sourceStream .mapValues(errorProneMapper) .branch(errorPredicate, successPredicate); int errorIndex = 0; int successIndex = 1; branched[errorIndex].to(errorReportTopic); branched[successIndex].mapValues(v -> ValueContainer::formatValue).to(outputTopic);
- 35. 35 Processing Exceptions DSL Predicate<String, ValueContainer> errorPredicate = (k, v) -> v.exception() != null; Predicate<String, ValueContainer> successPredicate = (k,v) ->!errorPredicate.test(k,v); KStream<String, ValueContainer>[] branched = sourceStream.mapValues(errorProneMapper). branch(errorPredicate,successPredicate); int errorIndex = 0; int successIndex = 1; branched[errorIndex].to(errorReportTopic); branched[successIndex].mapValues(ValueContainer::formatValue) .to(outputTopic);
- 36. 36 Processing Exceptions PAPI class ErrorProneProcessor extends AbstractProcessor<String, String> { @Override public void process(String key, String value) { ValueContainer valueContainer = new ValueContainer(); try { valueContainer.addValue(value.substring(0, 5)); context().forward(key, valueContainer, To.child(successProcessor)); } catch (Exception e) { context().forward(key, e.getMessage(), To.child(errorSink) } } }
- 37. 37 Processing Exceptions PAPI Topology topology = new Topology(); topology.addSource(sourceTopicNode, sourceTopic); topology.addProcessor(errorProneProcessor, ErrorProneProcessor::new, sourceTopicNode); topology.addProcessor(successProcessor, SuccessProcessor::new, errorProneProcessor); topology.addSink(errorSink, errorReportTopic, errorProneProcessor); topology.addSink(outputTopicNode, outputTopic, upperCaseProcessor);
- 38. 38 Uncaught Exceptions kafkaStreams.setUncaughtExceptionHandler( (thread, exception) -> { LOG.info(”It ran on my box fine!"); });
- 39. 39 StateRestoreListener kafkaStreams.setGlobalStateRestoreListener(new StateRestoreListener() { @Override public void onRestoreStart(TopicPartition topicPartition, String storeName, long startingOffset, long endingOffset) { } @Override public void onBatchRestored(TopicPartition topicPartition, String storeName, long batchEndOffset, long numRestored) { } @Override public void onRestoreEnd(TopicPartition topicPartition, String storeName, long totalRestored) { } });
- 40. 40 Kafka Streams StateListener kafkaStreams.setStateListener((newState, oldState) -> { if( newState == State.RUNNING && oldState == State.REBALANCING ) { // Do something here } else if ( newState == State.REBALANCING && oldState == State.RUNNING ) { // Do something else } });
- 41. 41 Wrapping Up • Sub-topologies, tasks and threads • Tasks and state • Failover
- 42. 42 Wrapping Up • General Error handing • Per-Record Error handling • Events
- 43. 43 Thanks! Kafka Streams in Action book signing @4:15 at the Confluent booth. Stay in Touch! • https://slackpass.io/confluentcommunity • https://www.confluent.io/blog/