10 essentials steps for kafka streaming services
4 likes744 views
The document discusses building event-driven microservices using Apache Kafka and Kafka Streams, outlining key features and benefits such as high throughput messaging and data locality. It emphasizes the importance of streaming as a toolset for handling data in flight, evolving service architectures, and ensuring fault tolerance. The role of events for both notification and data replication is highlighted as central to scalable and efficient service ecosystems.
10 essentials steps for kafka streaming services
- 1. 1 Building Event Driven Microservices with Apache Kafka and Kafka Streams Ben Stopford @benstopford
- 3. 3 Evolution!
- 4. 4 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
- 6. 6 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 7. 7 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 8. 8 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 9. 9 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 10. 10 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 11. 11 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 12. 12 Streaming == Manipulating Data in Flight
- 14. 14 EcosystemsApp Increasingly we build ecosystems
- 15. 15 SOA / Microservices / EDA Customer Service Shipping Service
- 16. 16 The Problem is DATA
- 17. 17 Most services share the same core facts. Catalog Most services live in here
- 18. 18 So how do we share data between services? Orders Service Shipping Service Customer Service Webserver
- 19. 19 Buying an iPad (with REST) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
- 20. 20 Buying an iPad with Events Message Broker (Kafka) Notification Data is replicated (incrementally) Submit Order Order Created Customer Updated Orders Service Shipping Service Customer Service Webserver KAFKA
- 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
- 22. 22 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
- 23. 23 Events have two hats Notification Data replication
- 24. 24 Events are the key to scalable service ecosystems
- 25. 25 Streaming is the toolset for dealing with events as they move!
- 26. 26 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 27. 27 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 28. 28 What is a Distributed Log?
- 29. 29 Shard on the way in Producing Services Kafka Consuming Services
- 30. 30 Each shard is a queue Producing Services Kafka Consuming Services
- 32. 32 Reduces to a globally ordered queue
- 35. 35 Build ‘Always On’ Services Rely on Fault Tolerant Broker
- 36. 36 Services can “Rewind & Replay” the log Rewind & Replay
- 37. 37 Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 38. 38 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 40. 40 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 41. 41 A database engine for data-in-flight
- 42. 42 SELECT card_number, count(*) FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; Continuously Running Queries
- 43. 43 Features: similar to database query engine JoinFilter Aggr- egate View Window
- 47. 47 Scales Out
- 49. 49 Streaming is about 1. Processing data incrementally 2. Moving data to where it needs to be processed (quickly and efficiently) On Notification Data Replication
- 50. 50 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 51. 51 Steps to Streaming Services
- 52. 52 1. Take Responsibility for the past and evolve
- 53. 53 Stay Simple. Take Responsibility for the past Browser Webserver
- 55. 55 2. Raise events. Don’t talk to services.
- 56. 56 Events have two hats Notification Data replication
- 57. 57 Raise events. Don’t talk to services Browser Webserver Orders Service
- 58. 58 KAFKA Order Requested Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
- 59. 59 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
- 60. 60 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Kafka as a Backbone for Events
- 61. 61 3. Use Connect (& CDC) to evolve away from legacy
- 63. 63KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use the Database as a ‘Seam’ Connect Products
- 64. 64 4. Make use of Schemas
- 65. 65KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Schemas to validate Backwards Compatibility Connect Products Schema Registry
- 66. 66 5. Use the Single Writer Principal
- 67. 67KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Apply the single writer principal Connect Products Schema Registry Order Completed
- 69. 69 Single Writer Principal - Creates local consistency points in the absence of Global Consistency - Makes schema upgrades easier to manage.
- 70. 70 6. Store Datasets in the Log
- 71. 71 Messaging that Remembers Orders Customers Payments Stock
- 72. 72 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service New Service, No Problem! Connect Products Schema Registry Order Completed Repricing
- 73. 73 Orders Customers Payments Stock Single, Shared Source of Truth
- 74. 74 But how do you query a log?
- 75. 75 7. Move Data to Code
- 76. 76
- 77. 77 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Materialize Stock ‘View’ Inside Service KAFKA
- 78. 78 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Take only the data we need KAFKA
- 79. 79 Data Movement Be realistic: • Network is no longer the bottleneck • Indexing is: • In memory indexes help • Keep datasets focused
- 80. 80 8. Use the log instead as a ‘database’
- 81. 81 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Apply Event Sourcing KAFKA Table
- 82. 82 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Order Service Loads Reserved Stocks on Startup KAFKA
- 83. 83 Kafka has several features for reducing the need to move data on startup - Standby Replicas - Disk Checkpoints - Compacted topics
- 84. 84 9. Use Transactions to tie All Interactions Together
- 85. 85 OrderRequested (IPad) 2a. Order Validated 2c. Offset Commit 2b. IPad Reserved Internal State: Stock = 17 Reservations = 2 Tie Events & State with Transactions
- 86. 86 Connect TRANSACTION Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Transactions KAFKA
- 87. 87 10. Bridge the Sync/Async Divide with a Streaming Ecosystem
- 88. 88 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service (see previous figure) Order Created Order Validated Orders View Q in CQRS Orders Service C is CQRS Services in the Micro: Orders Service Find the code online!
- 89. 89 Orders Customers Payments Stock Query Engine (Kstreams/KSQL) Larger Ecosystems HISTORICAL EVENT STREAMS
- 90. 90 Kafka KAFKA New York Tokyo London Global / Disconnected Ecosystems
- 91. 91 So…
- 92. 92 Good architectures have little to do with this:
- 93. 93 It’s about how systems evolves over time
- 94. 94 Request driven isn’t enough • High coupling • Hard to handle async flows • Hard to move and join datasets.
- 95. 95 Leverage the Duality of Events Notification Data replication
- 96. 96 With a toolset built for data in flight
- 97. 97 • Broadcast events • Retain them in the log • Evolve the event-stream with streaming functions • Recasting the event stream into views when you need to query. Event Driven Services
- 99. 99 Thank You @benstopford Blog Series: https://www.confluent.io/blog/tag/microservices/