#NetflixEverywhere Global Architecture
28 likes5,103 views
This document summarizes Netflix's journey to building a globally ubiquitous and failure-resilient architecture. It describes how Netflix evolved from a single data center architecture to a multi-region active-active design using microservices, Cassandra for data storage, EVCache for caching, and virtual DNS regions for traffic management. The architecture is designed to reliably serve customers from any region by replicating data and traffic across regions and implementing failover mechanisms.
#NetflixEverywhere Global Architecture
- 1. Josh Evans - Director of Operations Engineering March, 2016 #NetflixEverywhere Global Architecture
- 4. Outrage
- 5. Withdrawal
- 9. Never fail the same way twice Failure-Driven Architecture
- 10. #NetflixEverywhere Failure-Driven Architecture Never fail the same way twice
- 11. • Introductions • Failure-Driven Architecture • Taking It Global #NetflixEverywhere Our Talk Today
- 12. • Introductions • Failure-Driven Architecture • Taking It Global #NetflixEverywhere Our Talk Today
- 13. 1999 – 2009 • Ecommerce (DVD Streaming) 2009 – 2013 • Playback Services (Activate, Manifests, DRM) 2013 - present • Operations Engineering – CD, RTA, Chaos, Performance Josh Evans – Director of Operations Engineering jevans@netflix.com
- 14. Bringing movies & TV shows from all over the world to people all over the world • Streaming, on demand, subscription • Global & regional licensing • Hollywood, independent, international • Striving for global ubiquity
- 15. 2007 • Jan – Windows 2008 • May – Roku • Oct – LG, Samsung Blu-ray • Oct – Apple Mac • Nov – XBox 360 2009 • Jun – LG DTV • Nov –Sony PS3 (disc) • Nov – Sony Bravia – DTV & Blu-ray Device Ubiquity
- 16. 2011 • May – Android • Nov – First e-readers – Kindle Fire, Nook Device Ubiquity 2010 • Mar – Nintendo Wii (disc) • Apr – Apple iPad • Aug – Apple iPhone • Sep – Apple TV • Oct – Sony PS3 (no disc) • Oct – Nintendo Wii (no disc) • Nov – Windows Phone 7
- 19. 2010 - Canada 2011 - Latin America 2012 - UK, Ireland, Nordics 2013 – Netherlands 2014 - Austria, Belgium, France, Germany, Luxembourg, Switzerland 2015 - Australia, New Zealand, Japan, Spain, Italy, Portugal Geographic Ubiquity
- 20. • English • Spanish (Latin American) • Portuguese (Brazilian) • Dutch • French • German • Japanese • Spanish (Castilian) • Italian • Portuguese (European) Language Ubiquity - Subs, Dubs, UI
- 21. 75,000,000
- 22. • Introductions • Failure-Driven Architecture • Taking It Global #NetflixEverywhere Our Talk Today
- 23. August 2008
- 24. • No automation, virtualization, standardization • Manual, error prone, slow • Big iron & monoliths DC2 2009
- 26. US-East-1 Amazon Web Services 2010 • Scale & elasticity • Virtual, programmable • Global footprint
- 27. • Micro-services • Database • Cache • Traffic Architectural Pillars
- 28. • Micro-services • Database • Cache • Traffic Architectural Pillars
- 33. • Micro-services • Database • Cache • Traffic Architectural Pillars
- 34. NoSQL but… • Not web scale • Throttling Modest scale • 100s of play starts / second • 10,000s of requests / second • 10s of billions of records SimpleDB
- 35. • Micro-services • Database • Cache • Traffic Architectural Pillars
- 36. Ephemeral Volatile memCache (EVCache) Clustered memcached optimized for Netflix use cases
- 37. EVCache Server Memcached Prana (Sidecar) Monitoring & Other Processes Eureka Client Application Client Library EVCache Client Shards, consistent hashing TTLs & LRU EVCache Architecture
- 38. Zone A Client Application Client Library EVCache Client Zone B Client Application Client Library EVCache Client Zone C Client Application Client Library EVCache Client . . .. . .. . . Reads
- 39. Zone A Zone B Zone C . . .. . .. . . Writes Client Application Client Library EVCache Client Client Application Client Library EVCache Client Client Application Client Library EVCache Client
- 40. 1. Read from cache 2. On cache miss call service 3. Service calls DB & responds 4. Service updates cache Client Application Client Library EVCache Client Service Client S S S S. . . DB DB DB DB. . . . . . Fronting Micro-services . . .
- 41. Linear Scaling • 30 million requests/sec • 2 trillion requests per day globally • Hundreds of billions of objects • Tens of thousands of memcached instances • Milliseconds of latency per request
- 44. • Micro-services • Database • Cache • Traffic Architectural Pillars
- 45. US-East-1 EU-West-1 UK/IE, Nordics, Netherlands Latin America DNS Geo Mapping Canada US
- 46. • Micro-services • Database! • Caching • Traffic Architectural Pillars
- 47. Why Cassandra? • NoSQL at scale • Open source • Multi-region • Multi-directional • CAP Choices – Availability – Partition tolerance – Eventual consistency* Scalable, Durable, Global
- 48. Single Region, Multiple AZs 1. Client writes to any node 2. Coordinator replicates to nodes 3. Nodes ack to coordinator 4. Coordinator acks to client 5. Write to commit log Zone A Zone B Zone C Zone B Zone C Client Zone A • Hinted handoff to offline nodes Local Quorum (Typical) 100ms
- 49. Not quite fast enough
- 51. US-East-1US-West-2 EU-West-1 Isthmus Spring 2013 Survive a regional ELB outage
- 52. AZ1 AZ2 AZ3 US-EAST-1 ELBs Zuul Data Data Data Geo-located state/province AZ1 AZ2 AZ3 US-WEST-2 ELBs Zuul Data Data Data Americas Internet Traffic Eastern NA + LatAm Traffic • Zuul routes locally or remotely • Eureka - multi-region aware Isthmus
- 53. US-East-1US-West-2 EU-West-1 Active-Active 2013 - 2014 Survive a large-scale regional service outage
- 55. Region BRegion A Zone A Zone B Zone C Zone B Zone C Zone A Zone A Zone B Zone C Zone C Client Client Zone AZone B Multi-Region Writes 500ms Bi-directional Nightly compare & repair Local Quorum (Typical)
- 56. EVCache Replication Repl Writer SQS Application Client EVCache Replication Repl Writer 1. Set or delete 2. send metadata 3. poll msg 6. set or delete Application Client SQS 7. read EVCache Cross-Region Replication Region BRegion A
- 58. ELB US-West-2 ELB US-East-1 ELB EU-West-1 DNS api-global.netflix.com UltraDNS Route53
- 59. DNS api-global.netflix.com • Remove state from geo bucket ELB US-West-2 ELB US-East-1 ELB EU-West-1
- 60. api-global.netflix.com DNS • Remove state from geo bucket • Add state to geo bucket • Log event • For each end point ELB US-West-2 ELB US-East-1 ELB EU-West-1
- 66. • Introductions • Failure-Driven Architecture • Taking It Global #NetflixEverywhere Our Talk Today
- 69. Before Global • English • Spanish (Latin American) • Portuguese (Brazilian) • Dutch • French • German • Japanese • Spanish (Castilian) • Italian • Portuguese (European) Global • Chinese • Korean • Arabic Language Ubiquity
- 72. March 18th, 2016 Daredevil Season 2 All episodes, all devices, all countries Simultaneously Content Ubiquity
- 74. US-East-1US-West-2 EU-West-1 Reliably and efficiently serve any customer from any region Netflix Global 2015
- 77. Ubiquitous Data
- 78. EVCache Replication Repl Writer Application Client Kafka SQS • High latency • Read once Kafka • Low latency • Multiple readers • > 1M replications/sec
- 79. US Ring US Ring EU Ring EU-West-1US-East-1 1. Extend US ring to EU region & run repairs Client 2. Dual Write 3. Forklift
- 82. us-east-1-na • East US • East CA • MX us-west-2 • APAC • West US • West CA eu-west-1 • Europe • Mid East • Africa us-east-1-sa • LatAm • Not MX Virtual DNS Regions
- 83. • Fixed virtual modules • Origin tier • Standardized names api-global.netflix.com api-global.us-west-2 .prodaa.neflix.com api-global.us-east-1-sa .prodaa.neflix.com api-global.us-east-1-na .prodaa.neflix.com api-global.eu-west-1 .prodaa.netflix.com api-global.us-west-2.origin .prodaa.neflix.com api-global.us-east-1.origin .prodaa.neflix.com api-global.eu-west-1.origin .prodaa.neflix.com ELB ELB ELB Virtual Origin DNS Tiers
- 93. “Going global is just like having a baby.” - Reed Hastings, Netflix CEO
- 94. What’s next? • Global latency • Edge computing • ML-based monitoring • Self-healing systems • Capacity utilization • Fast, autonomous traffic • Integrate DB & caching #NetflixEverywhere
- 95. Takeaways
- 96. Never fail the same way twice Christmas Eve 2012 Today
- 97. Know your resiliency patterns Pattern Properties DC SPoF, infrastructure heavy lifting Cloud (one region) Multiple DCs, one control plane Islands Regional containment Isthmus Regional ELB bypass Active-active Regional failover Global Ubiquity, resiliency, efficiency
- 98. Invest in architectural pillars • Micro-services • Database • Caching • Traffic
- 99. #NetflixEverywhere Think globally, act locally
- 100. netflix.github.io
- 101. netflix.github.io
Editor's Notes
- #3: On Christmas eve, 2012 Netflix experiences a region-wide outage due an accidental ELB configuration change Many engineers were on call, missing time with their families They spent much of the night and into the morning trying to mitigate the impact of the outage on our customers but to no avail We ultimately had to wait for Amazon to address the root cause And our members, many of them new to Netflix were unable to stream Their responses varied in intensity from…
- #47: In preparation for UK launch we starting shoring up our data persistence story – one that enabled multi-region capabilities
- #93: \