Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019
7 likes2,797 views
The document discusses the importance of ensuring consistency in asynchronous data flows, emphasizing the model of 'eventual consistency' where each event must occur exactly once. It highlights the significance of mechanisms like dead letter queues and circuit breakers to handle errors and maintain system performance while ensuring a good user experience. Additionally, it underscores idempotence in event processing to prevent duplicate entries and manage system reliability effectively.
Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019
- 1. Iago Borges
- 3. Who are we?
- 4. Active credit card customers Active digital checking accounts Employees Engineers
- 6. Why is it so important?
- 7. That's why
- 8. Auditing
- 10. Whenever an event happens, the entire world must be eventually in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 11. Good eventual consistency is good enough
- 12. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 13. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 14. Impact of performance on consistency of asynchronous data flows
- 15. Whenever an event happens, the entire world must eventually be in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 16. "Anything that can go wrong will go wrong"
- 17. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 18. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 19. Impact of availability on consistency of asynchronous data flows
- 20. Maybe we just retry it until it works? ● Processing will be stuck. ● Hard to configure properly. ● Very hard observability.
- 21. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance Dead letter queue
- 22. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3 FTP PAYMENT FILE ARRIVED NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance How about this? Dead letter queue
- 23. Dead letters queue!! ● Fluid processing, regardless of failures. ● Great observability. ● Simpler configuration. ● One solution to rule them all.
- 24. How do we dead letter?
- 25. How do we dead letter? Random Service 2 Random Service 1 RANDOM EVENT TOPIC Mortician Produces to deadletter topic when an error is encountered AWS S3 Database DB references S3 Original message is replayed after problem is fixed HTTP The event is successfully propagated downstream No events lost
- 26. How do we make a good experience out of it?
- 27. How do we make a good experience out of it?
- 28. How do we make a good experience out of it?
- 29. So, would we keep killing letters non stop? Random Service 1 Random Service 2 Dead letter queue Service is failing multiple times Overloaded DB Doing a bunch of requests, overloading the database even more Producing a bunch of dead letters
- 30. Circuit breakers ● Protects your systems from DDOSing each other ● Once a lot of errors start happening it realises something is wrong and stop processing requests ● Probes eventually to know if things got back ● Self recovery
- 31. Circuit breakers Random Service 1 Failing service Dead letter queue Start throwing errors without even processing after multiple errors Overloaded DB Give some rest for the database to recover Stop sending unnecessary dead letters Circuit breaker
- 32. Circuit breakers Random Service 1 Failing service Dead letter queue Start working normally again once errors stop happening Overloaded DB Give some rest for the database to recover No need for deadletters nowCircuit breaker
- 33. Whenever an event happens, the world must be eventually in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 34. "The night is dark and full of retries" As it should
- 35. Idempotence "Every event shall be idempotent" Nubank Engineers, circa 2014
- 36. Datomic ● Serial writes ● Immutable ● Fact based. No fact is deleted or changed ● You can force to be in a specific state in time
- 37. Idempotence Random Service 1 Random Service 2 Sends event X Processes and persists event X Query: does it exists? No DB Writes
- 38. Idempotence Random Service 1 Random Service 2 Sends the same event X again Processes the same event X again Query: does it exists? Yes, it has just been written, right?? DB Doesn't write, because the query returned it already exists, right??
- 39. Idempotence Random Service 1 Random Service 2 Sends the same event X again Processes the same event X again Query: does it exists? Yes, it has just been written, right?? DB Doesn't write, because the query returned it already exists, right?? Well… Maybe...
- 40. Idempotence Random Service 1 Random Service 2 Publishes duplicated events with uniquely identifiable ID Processes duplicated event DB Attempts to write duplicate record, encounters unique ID conflict and throws exception Because it's serial!
- 41. Finagle ● Response classification ● Retry behavior ● Timeouts ● Load balancing ● Circuit breakers (fail accrual, fail fast, threshold fail detector) ● Connection pooling
- 42. What we have learned about consistency ● Clients expect companies to consistently take care of their money ● Errors will happen and they're not so bad, embrace and treat them well. ● Everything should be idempotent. ● Keeping things working well is awesome! (ORLY??)