Cache Rules Everything Around Me - DevIntersection - December 2022
- 1. C.R.E.A.M. get the memory (dollar dollar bill ya'll) August 2022 Matthew D. Groves | Product Marketing Manager aka “Passionate Bravo” Cache Rules Everything Around Me
- 2. 2 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
- 3. 3 The Need for Speed https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
- 4. 4 Cache Rules Everything Around Me Dollar dollar bill ya'll Android apps iPhone apps CPU cache CDNs Web browser DNS
- 5. 01/ 02/ 03/ 04/ 05/ 06/ What is caching? Use Cases Reading from Cache Writing into Cache Cache Invalidation Gotchas Agenda
- 6. 6 ho am I? • Matthew D. Groves • Microsoft MVP, author at Pluralsight, Manning, Apress • Product Marketing Manager at Couchbase • https://twitter.com/mgroves • https://www.linkedin.com/in/mgroves/ • C# Advent: https://csadvent.christmas/
- 7. 7 1 hat is Caching?
- 8. 8 Caching Collection of duplicated data to serve future requests faster. • First formally described by Maurice Wilkes in 1965 • "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper What is Caching? Key Data 1 Proteck Ya Neck 2 C.R.E.A.M. 3 Brooklyn Zoo 4 Criminology … … Key Data 2 C.R.E.A.M. 4 Criminology Records Cache
- 9. 9 Powerpoint as a Cache What is Caching? Key Value What year was Wu-Tang Clan Formed? 1992
- 10. 10 2 Caching Use Cases
- 11. 11 Caching Use Cases What is Caching used for?
- 12. 12 Caching Use Case 1: Faster Database Access • Databases can rely heavily on disk • Reading the same data over again could mean accessing the disk over again • Store frequently accessed data in a cache, to reduce pressure on database/disk
- 13. 14 Caching Use Case 1: Faster Database Access DATABASE APP LAN CACHE CACHE CACHE
- 14. 15 Caching Use Case 2: Faster Web Browsing • Any given URL could result in hundreds of files that are loaded over HTTP • A single JS file may be loaded by every page on a site. • Cache static assets (JS, images, sound, icons, etc) on a user's laptop to avoid loading them over the network every time
- 15. 16 Caching Use Case 2: Faster Web Browsing CACHE
- 16. 17 Caching Use Case 3: Efficient API Use • API usage may involve an internet call (with latency) • API calls may cost per use, or may be limited • Storing results in cache may help reduce latency AND reduce costs
- 17. 19 Caching Use Case 3: Efficient API Use API APP LAN/WAN CACHE CACHE CACHE
- 18. 20 Caching Use Case 4: Maintaining Availability • Retrieving data from other processes • Microservices • Latency is secondary concern
- 19. 21 Caching Use Case 4: Maintaining Availability CACHE
- 20. 22 Cache Rules Everything Around Me Dollar dollar bill ya'll Disconnected Availability Slow Repetition Constrained Costs
- 21. 23 3 Reading From Cache
- 22. 24 Using a Cache 101 Your Code Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo Cache Key Value A Protect Ya Neck B C.R.E.A.M. C Brooklyn Zoo D Criminology E Triumph F Uzi G Method Man H Wu-Tang Clan Ain't . . . . . . . . . Database 1. Lookup "A" 2. Lookup "B" 3. Lookup "C" HIT! HIT! MISS! 4. Lookup "C" again? WALK OF SHAME
- 23. 25 4 Writing Into Cache
- 24. 26 What about writes? • Data is duplicated, stored in cache and record • The system is responsible for keeping the cache updated • Commons methods: • write-through • write-around • write-back Stuff needs to get in the cache somehow
- 25. 27 write-through 1. System gets a new/updated piece of data 2. Write to cache and record "at the same time" 1. Write to record 2. If that didn't succeed, the write failed. 3. Write to cache 4. If that didn't succeed, the write failed. 5. If the system reaches this point, the write succeeded! Why or why not use this?
- 26. 28 write-around 1. System gets a new/updated piece of data 2. Write the data to the record ONLY 3. Invalidate the record in the cache Why or why not use this?
- 27. 29 write-back 1. System gets a new/updated piece of data 2. Write the data to the cache. 3. Update the record asynchronously (don't wait on it to finish) 4. If the cache write succeeded, then consider the write succeeded. Why or why not use this?
- 28. 30 Cache Writing Methods Method Write performance Write integrity Best for…? write-through ➖ ➕ High % reads write-around ➖ ➕ High % read later write-back ➕ ❔ Mixed reads / writes
- 29. 31 Couchbase A database with a built-in managed cache https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html
- 30. 32 Couchbase and Writing to the Cache await collection.InsertAsync("A", new { title = "Protect Ya Neck" }); await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options => { options.Durability(DurabilityLevel.None); // OR: options.Durability(DurabilityLevel.Majority); // OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive); // OR: options.Durability(DurabilityLevel.PersistToMajority); });
- 31. 33 Durability (with 3 replicas) None Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ⚠ ⚠ Node 4 ⚠ ⚠
- 32. 34 Durability (with 3 replicas) Majority Memory Disk Node 1 Node 2 Node 3 ✔ ⚠ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
- 33. 35 Durability (with 3 replicas) MajorityAndPersistToActive Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ⚠ Node 4 ✔ ⚠
- 34. 36 Durability (with 3 replicas) PersistToMajority Memory Disk Node 1 Node 2 Node 3 ✔ ✔ ⚠ ⚠ ✔ ✔ Node 4 ✔ ✔
- 35. 37 Latency Numbers Every Programmer Should Know https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315
- 37. 39 Powerpoint as a Cache Key Value What year was Wu-Tang Clan Formed? 1992 What was Wu-Tang's first album called? Enter the Wu-Tang What year was Maurice Wilkes born? 1913 What year did Maurice Wilkes die? 2010 What is the most recent Wu- Tang studio album? Once Upon a Time in Shaolin
- 38. 40 Cache Invalidation is Hard There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton How to decide what to invalidate (what data to "evict"): cache policy https://www.nndb.com/people/400/000031307/
- 39. 41 Cache Policy 1: LRU (Least Recently Used) Key How new? A 0 B 1 3 C 2 D 4 Cache max size: 3 Order of reads: A, B, C, B, D X ---------------------------------------------------------------------------------
- 40. 42 Cache Policy 2: NRU (Not Recently Used) Lower scores evicted (at random) Key Referenced? Modified? Score A 0 0 0 B 0 0 0 C 0 1 1 D 1 0 2 E 1 1 3
- 41. 43 Cache Policy 3: None Don't evict anything
- 42. 44 More Cache Policies • RR – random replacement • FIFO and LIFO – treat a cache like a queue • MRU – MOST recently used • LFU – least frequently used • Belady's algorithm – not possible to implement • Machine learning • https://en.wikipedia.org/wiki/Cache_replacement_policies
- 43. 45 6 Gotchas
- 44. 46 Gotcha 1: Cache Size • Problem: too small or too big cache • Symptom: High turnover and churn (many evictions) or long lifetimes (few evictions) • Solutions: • Monitoring • Size the cache • "Value" vs "Full" eviction
- 45. 47 Gotcha 2: "Close" Caching • Problem: running the cache on the same machine as the application • Symptom: redundancy, hot spots, inconsistency, availability • Solutions: • Use a distributed cache • Running on its own machine(s) / cluster • Couchbase, of course!
- 46. 48 Gotcha 3: Bad Eviction Policy • Problem: using the wrong eviction policy • Symptom: lots of overhead, cache checking • Solutions: • Don't Write an Eviction Policy (or better yet don't write a Cache) • Benchmarking: "hit ratio" and "latency" • "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio") • "Latency" – how long it takes from request to response (e.g. 10μs latency)
- 47. THANK YOU
- 48. 50 References • https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/ • https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback- caching-with-python • https://en.wikipedia.org/wiki/Maurice_Wilkes • https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf • https://stackoverflow.com/a/4087315/40015 • https://www.techtarget.com/searchstorage/definition/cache • https://en.wikipedia.org/wiki/Cache_replacement_policies • https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability • https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure- considerations.html#durable-writes