Kafka Summit 2020: If an event is published to a topic and no one is around to consume it, does it make a sound?
1 like917 views
The document discusses the relationship between event streams and state management in the context of workflow automation. It highlights the importance of translating stream events into meaningful actions and the complexities of managing state within these systems. The author emphasizes the interaction between declarative and imperative thinking when integrating streams with workflows.
![SELECT name
FROM people
WHERE city = `Berlin´
reader = new BufferedReader(new FileReader("people.txt"));
while ((row = reader.readLine()) != null) {
String[] data = row.split(",");
if ("Berlin".equals(data[3])) { // city
System.out.println(data[1]); // name
}
}
reader.close();
@berndruecker](https://image.slidesharecdn.com/doesaneventmakeasound-200714144721/85/Kafka-Summit-2020-If-an-event-is-published-to-a-topic-and-no-one-is-around-to-consume-it-does-it-make-a-sound-4-320.jpg)
![Functional programming, reactive & streaming = the greatest thing since sliced bread!
… the term Event Stream becomes widely
adopted and promotes a new mental model
of how the systems are implemented.
… the main abstraction in use is a
composition of [stateless] functions
performing transformation on immutable
events. The output of an overall system can
be modeled as the application of those
composed functions on the input stream.
https://medium.com/holisticon-consultants/from-stream-to-state-eb1cade5d403
@berndruecker](https://image.slidesharecdn.com/doesaneventmakeasound-200714144721/85/Kafka-Summit-2020-If-an-event-is-published-to-a-topic-and-no-one-is-around-to-consume-it-does-it-make-a-sound-7-320.jpg)
Kafka Summit 2020: If an event is published to a topic and no one is around to consume it, does it make a sound?
- 1. If an event is published to a topic and no one is around to consume it, does it make a sound? Bernd Ruecker Co-Founder and Chief Technologist of Camunda @berndruecker
- 2. Warning: This talk might contain nonsense @berndruecker
- 3. People that understand functional programming The others @berndruecker
- 4. SELECT name FROM people WHERE city = `Berlin´ reader = new BufferedReader(new FileReader("people.txt")); while ((row = reader.readLine()) != null) { String[] data = row.split(","); if ("Berlin".equals(data[3])) { // city System.out.println(data[1]); // name } } reader.close(); @berndruecker
- 7. Functional programming, reactive & streaming = the greatest thing since sliced bread! … the term Event Stream becomes widely adopted and promotes a new mental model of how the systems are implemented. … the main abstraction in use is a composition of [stateless] functions performing transformation on immutable events. The output of an overall system can be modeled as the application of those composed functions on the input stream. https://medium.com/holisticon-consultants/from-stream-to-state-eb1cade5d403 @berndruecker
- 8. Functions Immutable Events Event Streams State Input / Output …and actions in the real world… @berndruecker
- 9. Connor Riley #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. @berndruecker
- 10. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Stateless Stream Processor Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output → Job offer revoked @berndruecker
- 13. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Refine / change logic Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output @berndruecker
- 14. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output @berndruecker
- 16. #Cisco just offered me a job! Stateless Stream Processor Filter all tweets with #cisco tweets Filter tweets with similar content within 15 minutes time Stateful Stream Processor State is in the API! @berndruecker
- 17. #Cisco just offered me a job! Stateless Stream Processor Filter all tweets with #cisco tweets Filter tweets from our top community members Stateful Component (as stateless stream processor) State is in the component @berndruecker
- 18. #Cisco just offered me a job! tweets Enrich tweet with personal information of sender lookup Bad, because - Load on API - Availability - Other disadvantages of synchronous remote communication @berndruecker
- 19. #Cisco just offered me a job! tweets Person stream Stateful Stream Processor Enrich tweet @berndruecker
- 20. #Cisco just offered me a job! tweets Stateful Stream Processor Persons that tweeted Enrich tweet Gather personal info @berndruecker
- 21. #Cisco just offered me a job! tweets Stateful Stream Processor THIS is workflow logic! Workflow engines can hold state Enrich tweet Gather personal info @berndruecker Persons that tweeted
- 22. #Cisco just offered me a job! tweets Stateful Stream Processor Workflow engines can hold state Enrich tweet Gather personal info Publish / subscribe Mitigates problems of synchronous communication! @berndruecker
- 23. Example workflowClient.newWorker() .jobType("person-lookup") .handler((jobClient, job) -> { System.out.println("Gather personal information"); jobClient.newCompleteCommand(job.getKey()).send().join(); }).open(); publish / subscribe @berndruecker client.newCreateInstanceCommand() .bpmnProcessId("enrich-tweet") .variables(...) .latestVersion() .send().join(); client.newCreateInstanceCommand() .bpmnProcessId("order-process") .latestVersion() .send() .join(); Or use the Zeebe Kafka Connector
- 24. Declarative Thinking Imperative Thinking Or is it just me? @berndruecker
- 25. Berlin, Germany http://berndruecker.io/ mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
- 26. Where do we stand now? State goes into Stream Processor API Component Workflow Engine* *helps to implement common patterns in stateful components @berndruecker
- 27. Where do we stand? State goes into Stream Processor API Component Workflow Engine This translation requires additional thought! @berndruecker
- 28. Let‘s do another example: Vehicle Maintenance
- 29. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Call driver to stop and inspect … @berndruecker
- 30. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor @berndruecker
- 31. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Custom Connector with own state WorkflowsEvent Streams @berndruecker
- 32. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Custom Connector @berndruecker
- 33. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Deduplicate Messages Custom Connector @berndruecker
- 34. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 666555444333222 Asset: Bus_4736_Motor Time: 1593611320 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Semantic Aggregation Custom Connector @berndruecker
- 35. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical_for_too_long CorrelationKey: Bus_4736_Motor_oil_pressure_... Custom Connector Time windowing is done in stream processing @berndruecker
- 36. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 777666555444333 Asset: Bus_4736_Motor Time: 1593900001 Insight: oil_pressure_normal CorrelationKey: Bus_4736_Motor_oil_pressure_normal Semantic Aggregation STARTNEW WORKFLOW STARTOR KEEP WORKFLOW CLOSE WORKFLOW Custom Connector @berndruecker
- 38. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision WorkflowsEvent Streams @berndruecker Custom Connector
- 39. ActionInsightsMeasurements generates Leads to Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Custom Connector WorkflowsEvent Streams This regularly changed and the stream of events was replayed @berndruecker
- 40. Connecting two worldsCustom Connector - Deduplication - Semantic Aggregation - Start, correlate to or cancel workflow instances - Handle replays - … Streaming Workflow @berndruecker
- 42. Summary • Streams need to lead to actions outside the control of the streams architecture (otherwise nothing has really happened) • Where to handle state is a difficult question • Streams vs. Workflow might be about declarative vs. imperative thinking (what does your team understand?) • But streams and workflows also go hand in hand. Getting from stream to workflow involves a translation component @berndruecker