The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit London 2019
1 like1,622 views
The document discusses the challenges of implementing event-driven caching, particularly focusing on the complexities of cache invalidation and data consistency. It provides architectural insights on optimizing cache interactions using Kafka for change notifications and offers guidance on when to apply certain caching techniques. Key considerations include the latency of data retrieval, the size of cached data, and strategies for maintaining cache reliability.
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
Store the Kafka Topic in a Hash Table
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-48-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
In RAM Inside Your Application
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-49-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
Too Big for RAM? Store it on Disk
App
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-50-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
A Cache in Every Instance
[ ]Cache
[ ]Cache
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-51-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
Out of Process Cache
App App App
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-52-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-53-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
Shard Your Data
A-I J-R S-Z
[ ]Cache
[ ]Cache
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-55-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
How Does the Application Find its Data?
App
[ ]Cache
[ ]Cache
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-56-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
Caches can Forward Requests
[ ]Cache
[ ]Cache
[ ]Cache
App](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-61-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
Use a Proxy
AppProxy
[ ]Cache
[ ]Cache
[ ]Cache](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-62-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-64-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]
[ ] [ ]](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-65-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]
[ ] [ ]
?](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-66-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-67-320.jpg)
![© 2019 Bloomberg Finance L.P. All rights reserved.
© 2019 Bloomberg Finance L.P. All rights reserved.
[ ]
[ ]
[ ]](https://image.slidesharecdn.com/brendanpowersbloombergthepowerofeventdrivencaches-190520220337/85/The-Power-of-Event-Driven-Caches-Brendan-Powers-Bloomberg-L-P-Kafka-Summit-London-2019-68-320.jpg)
The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit London 2019
- 1. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. The Power of Event-Driven Caches Why caching is hard, and what you can do about it. Kafka Summit London May 13, 2019 Brendan Powers Software Engineer @brendan0powers
- 2. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Who am I?
- 3. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
- 4. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Database Access App DB
- 5. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Cache Access Patterns App Cache DB
- 6. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. “Thundering Herd” Problem DBApp Cache
- 7. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. “Thundering Herd” Problem DB
- 8. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. “Thundering Herd” Problem
- 9. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Concurrency App App ? Cache
- 10. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Is the Data in My Cache Valid? DBApp Cache ???
- 11. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Is the Data in My Cache Valid? ?? ?
- 12. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Your Database is the Source of Truth DB
- 13. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Push Writes to the DB into the Cache Cache DB
- 14. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Revised Architecture Diagram Cache App DB
- 15. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Cache Instances are Consistent DB Cache Cache Cache
- 16. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. How do I get Change Notification from my DB? Cach e DB
- 17. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Dual Writes App DB Cache Kafka
- 18. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Keep the Number of Write Paths Small ! DB Cache Kafka
- 19. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Dealing With Partial Write Failure DB App Cache Kafka
- 20. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Change Data Capture (CDC) DB App Write Log
- 21. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Change Data Capture (CDC) DB Write LogCache Kafka App
- 22. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Why Kafka? ● Durable ● Scalable ● Debuggable
- 23. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka’s Topic Offset Tracking 98765 DB Cache 4321
- 24. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Too Complicated? Cache App DB
- 25. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Too Complicated? DB Cache App
- 26. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Recent Writes User 1User 2User 1User 3User 2
- 27. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... . . . User 1User 2User 1User 3User 2
- 28. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 1User 2User 1User 3User 2
- 29. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 1 User 2 User 1 User 3User 2
- 30. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 1 User 2 User 1 User 3 User 2
- 31. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 2 User 1 User 3 User 2
- 32. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 1 User 3 User 2
- 33. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Never Forgets... User 1User 3User 2
- 34. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Full Records User 1{ “Id”: 492343, “name”: “User 1”, “Date_joined”, “05/05/1995”, …... } User 2
- 35. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. DB
- 36. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Cache
- 37. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Cache DB
- 38. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Simplifying Our Design DB Cache App
- 39. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Simplifying Our Design Cache App DB
- 40. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Fault Tolerant Cache App
- 41. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. “Thundering Herd” Problem
- 42. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. “Thundering Herd” Problem DB
- 43. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Scale Independence DB Cache Cache Cache Cache Cache
- 44. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. More Sleep, Less Phone Calls App Cache DB
- 45. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. When to Use This Technique ● When your cache needs to be up to date ● When latency is very important ● When you need to give clients flexibility in their caching strategy
- 46. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. When NOT to Use This Technique ● When your data rarely changes ● When you data size is very large (100s of GB or TB)
- 47. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. How? Cache ?
- 48. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Store the Kafka Topic in a Hash Table [ ]Cache
- 49. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. In RAM Inside Your Application [ ]Cache
- 50. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Too Big for RAM? Store it on Disk App [ ]Cache
- 51. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. A Cache in Every Instance [ ]Cache [ ]Cache [ ]Cache
- 52. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Out of Process Cache App App App [ ]Cache
- 53. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ]Cache
- 54. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. CACHE!
- 55. © 2019 Bloomberg Finance L.P. All rights reserved. Shard Your Data A-I J-R S-Z [ ]Cache [ ]Cache [ ]Cache
- 56. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. How Does the Application Find its Data? App [ ]Cache [ ]Cache [ ]Cache
- 57. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
- 58. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Questions?
- 59. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Kafka Connectors ● Debezium ● JDBC ● Dedicated connectors for MS SQL, Oracle, etc.
- 60. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Will My Data Fit in RAM? ● How much RAM do you have? ● How big are your records? ● Do you have to cache the entire record?
- 61. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Caches can Forward Requests [ ]Cache [ ]Cache [ ]Cache App
- 62. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. Use a Proxy AppProxy [ ]Cache [ ]Cache [ ]Cache
- 63. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved.
- 64. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ]
- 65. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ] [ ] [ ]
- 66. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ] [ ] [ ] ?
- 67. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ]
- 68. © 2019 Bloomberg Finance L.P. All rights reserved. © 2019 Bloomberg Finance L.P. All rights reserved. [ ] [ ] [ ]