Event Streaming Architecture - Deep Dive
- 1. EVENT STREAMING ARCHITECTURE - DEEP DIVE KARTHIKEYAN VK @KARTHIK3030
- 3. PROBLEMS WITH CURRENT ANALYTICS Not Realtime ETL was created during a period of monolithic architectures, data warehouses and relational databases.
- 4. PROBLEMS WITH CURRENT MICROSERVICES ARCHITECTURE • Client more details about our service • Too many details to be exposed • Chatty communication
- 5. PROBLEMS WITH CURRENT MICROSERVICES ARCHITECTURE • Gateway can take care of authentication. • Rate limiting possible • Things seems ok.
- 8. CHANGES ARE STILL HARD • Adding a new service needs to communicated to other services of its presence. • Chatty communication between services • REST+JSON is SLOW • GRPC better, but not a complete solution • Adding services adds new changes to the other services.
- 10. COMPARISON
- 12. ONLINE SHOP 1. No Knowledge of other services 2. Communication at services will. 3. Rest calls reduced. 4. Broadcast your events and can be ignored by other services or consumed.
- 13. ONLINE SHOPPING USING R.E.S.T 1.Order should know about shipment services 2.Shipment should get details of the customer from the customer service 3.What if there is customer details are changed ?
- 14. EVENT ARE BOTH FACTS AND TRIGGERS
- 15. PRODUCT ORDERED - FACT 1.Order submitted 2.Event sent to Kafka 3.Shipment reacts to the order being placed 4.No REST communication between order and shipment 5.Product updates itself based on process result
- 16. ADDRESS CHANGED - TRIGGER 1.Customer changes address 2.Event sent to Kafka 3.Shipment reacts to changes 4.Order does not have to care about any changes once it is placed. 5.Shipment updates its database 6.Other services will not react if it is not pertained to them, IGNORE
- 17. DEMO AZURE EVENT HUBS - SETUP
- 19. BOOKS TO READ
- 20. BASICS OF EVENT STREAMING • Databases vs streams • Command vs Events • Immutable Events • State, Streams and Immutability • Stream Joins
- 21. DATABASE VS STREAMS • Concept wise it is same • Every action takes places is stored as logs • Restore happens from reading the logs again and creating the database
- 22. COMMAND VS EVENTS VS IMMUTABLE EVENTS • Any request is a command. • After integrity check it becomes an Event. • Booking an air ticket is command, if there is ticket available, it becomes an event. It also becomes a fact. • When the ticket is cancelled, it is still an event, that is immutable and recorded in our system.
- 23. STATE, STREAMS AND IMMUTABILITY • When the ticket is booked, the State is saved. • When the ticket date is changed and ticket class is upgraded, there is a Stream of events that changes the state. • Ticket can even be cancelled but the history of the ticket booking will remain and recorded as logs and each event is Immutable.
- 24. STREAM JOINS • Joining different datasets to perform analysis • Stream-Stream Join • Stream-Table Join • Table-Table Join
- 25. STREAM-STREAM JOINS • User searches for a book and he clicks the search result. • You need to map the book searched and clicked book. • Now the search is recorded as one event and click is another event. • If you want to analyse the click through rate you need to bring together the search and click action, which can be connected by Session ID
- 26. STREAM-TABLE JOINS • User searches for a book • You need to show some recommendation based on his search • Now you need to join the search which is an event and his search history which is saved as table. • You can copy the table to local cache and join the search. But cache can become stale. Better is to subscribe the cache to the stream of user history
- 27. DEMO AZURE EVENT HUBS – ANOMALY DETECTION ON STREAMING DATA
- 28. REFERENCES • https://www.infoq.com/presentations/microservices-streams-state- scalability/ • https://docs.microsoft.com/en-in/azure/event-hubs/event-hubs-about • https://docs.microsoft.com/en-in/azure/event-hubs/event-hubs-tutorial- visualize-anomalies • https://github.com/Azure/azure-event- hubs/tree/master/samples/DotNet/Azure.Messaging.EventHubs
- 29. Q&A