EVCache Builderscon
- 1. Caching at Netflix: The Evolution of EVCache Scott Mansfield Builderscon 2018
- 12. 90 seconds
- 13. What do caches touch? Video production* Search* Choosing a profile My List Personalization* Loading home page* CDN selection* A/B tests Video encoding selection Choosing a profile Viewing title details* Playing a title* Subtitle / language prefs Localization strings Thumbs up / down Picking liked videos Video history* UI strings Session management Video bookmarks Device history Device management Authentication Scrolling home page* Title image selection Geographic location Signing up* * multiple caches involved
- 16. Key-Value store optimized for AWS and tuned for Netflix use cases Ephemeral Volatile Cache
- 17. What is EVCache? Distributed, sharded, replicated key-value store Based on Memcached Tunable in-region and global replication Resilient to failure Topology aware Linearly scalable Seamless deployments
- 18. Why Optimize for AWS Instances disappear Zones fail Regions become unstable Network is lossy Customer requests bounce between regions Failures happen and we test all the time
- 19. EVCache Use @ Netflix 3 regions 4 engineers ~100 clusters ~160 servers/cluster ~16,000 servers ~5,000,000 replications/sec ~100,000,000 ops/sec ~400,000,000,000 items ~6,000,000,000,000 ops/day ~3,000,000,000,000,000 bytes
- 20. What We're Covering 1) EVCache intro 2) Architecture 3) Use Cases 4) End to End 5) Supporting Features 6) Server Evolution 7) Optimization
- 21. Architecture
- 23. Architecture us-west-2a us-west-2cus-west-2b ClientClient Client Copy 1 Copy 3Copy 2
- 25. Writing (set, delete, add, etc.) us-west-2a us-west-2cus-west-2b ClientClient Client
- 26. Use Cases
- 27. Use Case 1: Lookaside Cache Application Client Library Client HTTP / gRPC Client S C Data Flow
- 28. Use Case 2: Transient Data Store Application Client Library Client Application Client Library Client Application Client Library Client Time
- 29. Use Case 3: Primary Store Offline / Nearline Precomputes for Recommendations Online Services Offline Services Online Application Client Library Client Data Flow
- 30. Online Services Offline Services Use Case 4: Versioned Primary Store Offline Compute Online Application Client Library Client Data Flow Dynamic Properties Control System (Valhalla)
- 31. Use Case 5: High Volume && High Availability Data Flow Application Client Library In-memory Remote HTTP/gRPC Client Compute & Publish on schedule S C Optional
- 32. Use Case 6: Personalization Fact Store Playlist Video History Personalization Experimentation PVRPreferences GPS https://vimeo.com/274954151
- 33. End to End
- 34. Pipeline of Personalization Compute A Compute B Compute C Compute D Online Services Offline Services Compute E Data Flow Online 1 Online 2
- 35. Full Circle Online Services Unified Logging Personalization Experimentation
- 37. Supporting Features Global data replication Polyglot clients Cache warming
- 38. Netflix Global Architecture eu-west-1 us-west-2 us-east-1 NA LATAMAPAC EMEA
- 39. Region BRegion A APP APP Repl Proxy Repl Relay 1 mutate 2 send metadata 3 poll msg 5 https send m sg 6 mutate4 get data for set Kafka Repl Relay Kafka Repl Proxy Cross-Region Replication 7 read
- 40. Polyglot Clients APP Java Client APP Prana Local Proxy Memcached APP Memcached Proxy HTTP APP HTTP Proxy
- 41. Cache Warming (Deployment) Cache Warmer Application Client Library Client Control S3 Data Flow Metadata Flow Control Flow
- 42. Server Evolution
- 44. Original Server ● Stock Memcached and EVCar (sidecar) ● All data stored in RAM in Memcached ● Expensive with global expansion / N+1 architecture Memcached EVCar external
- 45. Moneta Server V2
- 46. Moneta The Goddess of Memory ● Global data in cache means many copies ● Access patterns are heavily region-oriented ● Keep hot data in RAM, cold data on SSD
- 47. Moneta Server ● Adds Rend proxy and Mnemonic disk storage ● Still looks like Memcached from the outside Rend EVCar Memcached (RAM) Mnemonic (SSD) external internal
- 48. Rend ● High-performance Memcached proxy ● Manages connections, request orchestration, and communication ● Low-overhead metrics library ● Multiple orchestrators ● Parallel locking for data integrity ● Efficient connection pool Server Loop Request Orchestration Backend Handlers M E T R I C S Connection Management Protocol
- 49. Mnemonic ● Manages data storage on SSD ● Maps Memcached ops to RocksDB ops ● Sharded RocksDB in FIFO mode Rend Server Core Lib (Go) Mnemonic Op Handler (Go) Mnemonic Core (C++) RocksDB (C++)
- 50. Moneta in Production ● Held all of our personalization data ● One port for standard users (read heavy) ● Second port for async replication and batch users ● Maintains working set in RAM ● Optimized for precomputes ○ Replaces in-use data in L1 external internal EVCar Memcached (RAM) Mnemonic (SSD) Std Batch
- 51. Problem: Latency numbers higher than expected Evidence: 99.9th percentile of 9.68 ms Bug: Measuring time between requests, not of each request Solution: Properly instrument to get real timing "Optimization" #1
- 52. Fix: Batching and multiplexing requests Problem: High CPU load during peak traffic Evidence: Profiling shows CPU used on read/write syscalls Optimization #2 Rend Memcached 1R 6W 3R 4W 2W 5R
- 53. Rend Batching Backend ● Batch requests to Memcached ● Multiplex on connection pool ● Synchronous interface, asynchronous implementation
- 54. Relay 1 2 3 ... N Client connections 1 ... M Memcached connections Overview Sync Pool Async
- 55. Handler Wait for responseSubmit request Relay Client conn Client Connection
- 56. Relay 1 2 3 ... N Client connections Memcached connections rand() 1 ... M M+1 Monitor Read conn stats Wait Add? Add No Yes New Conn Relay
- 57. Pooled Connection relay Batcher Req Chan Reader Response Channels Map Chan Batch until full or timeout Write batch Data Map (req→res chan) Read responses Match Client connections External connection to Memcached Batching
- 58. Optimization #2 Problem: High CPU load during peak traffic Evidence: Profiling shows CPU used on read/write syscalls Fix: Batching and multiplexing requests Result: >15% reduction in CPU usage
- 59. Get Percentiles: ● 50th : 79.3 μs (76.6 μs) ● 75th : 92.1 μs (84.8 μs) ● 90th : 112 μs (101 μs) ● 95th : 140 μs (121 μs) ● 99th : 424 μs (378 μs) ● 99.5th : 511 μs (431 μs) ● 99.9th : 1.28 ms (651 μs) Moneta Performance in Prod Set Percentiles: ● 50th : 68.7 μs (59.5 μs) ● 75th : 82.3 μs (71.8 μs) ● 90th : 98.2 μs (84.3 μs) ● 95th : 107 μs (92.7 μs) ● 99th : 143 μs (118 μs) ● 99.5th : 169 μs (131 μs) ● 99.9th : 1.97 ms (264 μs) Latencies: peak (trough)
- 60. Moneta Performance in Prod
- 61. Back to the Future Server V3
- 62. Memcached Extstore Fantastic improvement to store data on disk, index in RAM Addition by Dormando, maintainer of Memcached Get rid of ~18 months * 2 engineers work No maintenance of proprietary code Everyone gets improvements (OSS) https://memcached.org/blog/nvm-caching/ https://github.com/memcached/memcached/wiki/Extstore
- 63. Advantages over Moneta Use 80 - 90% of disk instead of only 50% Single process Spectre & Meltdown made syscalls significantly slower Easier management Lower latency Cache warming possible for disk-backed clusters Enables disk for high write volume and multi-petabyte clusters
- 64. Deployment Nov 2017 Test extstore beta / bug fixes Dec 2017 First production deployment Feb 2018 Extstore fully deployed Apr 2018 New use cases start use
- 68. Backup (and Restore) Cache Warmer (Spark) Application Client Library Client Control S3Data Flow Control Flow
- 69. Lost Instance Recovery Cache Warmer (Spark) Application Client Library Client S3 Partial Data Flow Metadata Flow Control Flow Control Zone A Zone B Data Flow