Caching 101: Caching on the JVM (and beyond)
0 likes52 views
This document summarizes a presentation on caching 101. It discusses caching theory, patterns, and scaling approaches. It covers caching on the JVM using JSR 107 and different caching strategies like cache-aside, cache-through, and write-behind. It also discusses scaling caches by moving data off-heap, clustering caches across multiple machines, and achieving different levels of consistency like strong, eventual, and probabilistically bounded staleness.
Caching 101: Caching on the JVM (and beyond)
- 1. @alexsnaps @ljacomet#Devoxx #cache101 Caching 101 Caching on the JVM and beyond alex snaps - principal software engineer at terracotta louis jacomet - lead software engineer at terracotta
- 2. @alexsnaps @ljacomet#Devoxx #cache101 Agenda •Theory •Caching patterns •Scaling
- 3. @alexsnaps @ljacomet#Devoxx #cache101 Introductory Poll •Who knows nothing about caching? •Who already uses caching in production? •Who had caching related problems in production? •Who is interested in advanced caching patterns?
- 4. @alexsnaps @ljacomet#Devoxx #cache101 (Caching) theoryhttp://blog.spylight.com/sheldons-big-bang-theory-style/
- 5. @alexsnaps @ljacomet#Devoxx #cache101 There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton
- 6. @alexsnaps @ljacomet#Devoxx #cache101 CPU Caches https://en.wikipedia.org/wiki/Non-uniform_memory_access
- 7. @alexsnaps @ljacomet#Devoxx #cache101 Cache coherence • P write A to X, P read X => returns A when no writes happened in between, must be valid for single processor too • P2 writes A to X, P1 reads X => returns A if “enough time” has elapsed • P1 writes A to X, P2 writes B to X => no one can ever see B then A at X
- 9. @alexsnaps @ljacomet#Devoxx #cache101 Numbers explained L1 cache reference 1 s 2 heartbeats Branch mispredict 3 s Yawn L2 cache reference 4 s (Longer) Yawn Mutex lock / unlock 17 s Coffee preparation Main memory reference 100 s Brushing your teeth Send 2k over commodity network 4.2 s A song
- 10. @alexsnaps @ljacomet#Devoxx #cache101 Numbers explained Compress 1kB with zippy 33 m Sitcom episode Read 1MB from memory 2 h Bad commute SSD random read 4 h Half day at work Read 1MB from SSD 2.6 d Long week-end Round trip in datacenter 10 d Two weeks delivery Packet roundtrip CA to Be 4.8 y PhD thesis
- 12. @alexsnaps @ljacomet#Devoxx #cache101 Amdhal’s law •Parallelisation •P: parallel proportion •N: processor count
- 13. @alexsnaps @ljacomet#Devoxx #cache101 Amdhal’s law (cont’d) •Sequential •p: times a part was sped up •f: fraction of time not improved
- 14. @alexsnaps @ljacomet#Devoxx #cache101 Further reading • Gustafson’s law states that computations involving arbitrarily large data sets can be efficiently parallelized. • As computing powers grow, larger problems can be solved in a given time
- 15. @alexsnaps @ljacomet#Devoxx #cache101 Where and when to use a cache? https://www.flickr.com/photos/wwarby/ It depends … Measure, do not guess!
- 16. @alexsnaps @ljacomet#Devoxx #cache101 What is a cache in an application? • Data structure holding a temporary copy of some data • Trade off between higher memory usage for reduced latency • Targets: • Data which is reused • Data which is expensive to compute or retrieve
- 17. @alexsnaps @ljacomet#Devoxx #cache101 Desired cache features • Capacity control / eviction • Data freshness / expiry • Data consistency / invalidation • Fault tolerant
- 19. @alexsnaps @ljacomet#Devoxx #cache101 JSR 107 • Java Community Process driven standard • Specifies API and semantics for temporary, in-memory caching of Java objects, including object creation, shared access, spooling, invalidation, and consistency across JVM's
- 20. @alexsnaps @ljacomet#Devoxx #cache101 javax.cache API • CacheManagers managing … Caches ! • Expiry • Integration • Cache Entry Listeners • Entry Processors • Caching Annotations • Some Management
- 21. @alexsnaps @ljacomet#Devoxx #cache101 CacheManager • Caching.getCachingProvider() static method •CachingProvider.getCacheManager() • CachingProvider • CacheManager repository • Identified by name & ClassLoader
- 22. @alexsnaps @ljacomet#Devoxx #cache101 CacheManager •CacheManager • Manages Cache lifecycle: • Create & Destroy • Repository of (named) Cache instances • Manages Cache JMX (stats & management) • … is .unwrap()’ able
- 23. @alexsnaps @ljacomet#Devoxx #cache101 Cache aside - miss Application Cache Database
- 24. @alexsnaps @ljacomet#Devoxx #cache101 Cache aside - hit Application Cache Database
- 26. @alexsnaps @ljacomet#Devoxx #cache101 Cache aside conclusions • Solution most seen out there: Spring, Play, Grails, … • Most often based on annotations • Tricky to get the concurrency and / or atomicity right • Especially when rolling your own • Does not resolve doing multiple real invocations when warming cache
- 27. @alexsnaps @ljacomet#Devoxx #cache101 Hibernate boat1 = session.get( Boat.class, id ); boat2 = session.get( Boat.class, id ); boat1 == boat2 ?
- 28. @alexsnaps @ljacomet#Devoxx #cache101 Hibernate 2nd level cache • Stores “dehydrated” entity states • hibernate.cache.use_structured_entries • 4 strategies • ReadOnly • ReadWrite • NonStrictReadWrite • Transactional
- 29. @alexsnaps @ljacomet#Devoxx #cache101 Cache through - miss Application Cache Database
- 30. @alexsnaps @ljacomet#Devoxx #cache101 Cache through - hit Application Cache Database
- 31. @alexsnaps @ljacomet#Devoxx #cache101 Cache through - write Application Cache Database
- 33. @alexsnaps @ljacomet#Devoxx #cache101 Cache through conclusions • Requires different abstraction / modelling • Viewing the system of record through the cache may not be easy • Provides better guarantees and consistency as invalidation is no longer required • Still falls apart if data is modified by other applications / processes • Cost of writing is paid by thread putting in the cache
- 34. @alexsnaps @ljacomet#Devoxx #cache101 Application Cache Database queue Write behind
- 35. @alexsnaps @ljacomet#Devoxx #cache101 Application Cache Database queue Write behind - eviction (K1,V1) (K1,V1)
- 36. @alexsnaps @ljacomet#Devoxx #cache101 Write behind - failure Application Cache queue Database
- 37. @alexsnaps @ljacomet#Devoxx #cache101 Write behind - failure queue Database
- 39. @alexsnaps @ljacomet#Devoxx #cache101 Write behind conclusions • Scales out your writes • Batching and coalescing • Persistent queue or not • Idempotent operations are important in distributed and failure conditions
- 40. @alexsnaps @ljacomet#Devoxx #cache101 Refresh ahead Application Cache Database
- 41. @alexsnaps @ljacomet#Devoxx #cache101 Refresh ahead Application Cache Database
- 42. @alexsnaps @ljacomet#Devoxx #cache101 Scheduled refresh Application Cache Database
- 43. @alexsnaps @ljacomet#Devoxx #cache101 Scheduled refresh Application Cache Database
- 44. @alexsnaps @ljacomet#Devoxx #cache101 Scaling your cache
- 45. @alexsnaps @ljacomet#Devoxx #cache101 Moving off heap • JVMs have issues with large heaps due to garbage collection • OffHeap is a nice alternative for data which has a well known lifecycle • Perfect match for cache data • Performance hit due to binary representation compared to objects on heap
- 46. @alexsnaps @ljacomet#Devoxx #cache101 Want to know more? Talk on Thursday at 5.50pm room 10 by Chris Dennis (Terracotta)
- 47. @alexsnaps @ljacomet#Devoxx #cache101 Clustering • Cache shared between multiple machines • Different topologies • Peer to peer • Client server - with or without client cache • Consistency becomes a much harder problem
- 48. @alexsnaps @ljacomet#Devoxx #cache101 Terracotta clustering
- 49. @alexsnaps @ljacomet#Devoxx #cache101 Strong consistency Cache Terracotta client Terracotta server Terracotta client Terracotta client
- 50. @alexsnaps @ljacomet#Devoxx #cache101 Eventual consistency Cache Terracotta client Terracotta server Terracotta client Terracotta client
- 51. @alexsnaps @ljacomet#Devoxx #cache101 Probabilistically Bounded Staleness • In how “long” will a write be “readable” by all ? • How many “old versions” are still around ? • Depending on • Network delay • Node processing time • … delayed replication (e.g. batching) • LinkedIn’s data stores returned consistent data 99.9 percent of the time within 13.6 ms, and on SSDs (solid-state drives) within 1.63 ms.
- 52. @alexsnaps @ljacomet#Devoxx #cache101 Conclusion • You know it all now! • Don’t be afraid of caching • Consider it early on • Topologies! • Thanks! Questions?