Scaling the Netflix API

Editor's Notes

• #4: For Netflix, our API strategy can be discussed in the form of an iceberg. The public API strategy is the prominent, highly visible part of the iceberg that is above water. It is also the smallest part of the iceberg, in terms of mass. Meanwhile, the large mass of ice underwater that you cannot see is the most critical and biggest part of the iceberg. The visible part of the iceberg represents the public API and the part underwater is the internal strategy. Netflix’s strategy over the years has shifted from public to internal.
• #5: There are many ways to think of “scale”…
• #6: People generally think of scaling as growth in traffic…
• #7: And traffic growth typically equals server growth. System scaling requires a balance between need and capacity.
• #8: As a result, as traffic grows over time, scaling effectively requires adding capacity to support it.
• #9: To have an effective engineering organization, you need to scale in a variety of ways, not just in your systems. This presentation discusses these scaling needs. Of course, I will focus a bit on systems, but that is not the only area that requires focus to be successful.
• #11: Netflix’s focus is on providing the best global streaming video experience for TV shows and movies.
• #12: We now have more than 36 million global subscribers in more than 40 countries.
• #13: Those subscribers consume more than a billion hours of streaming video a month which accounts for about 33% of the peak Internet traffic in North America.
• #14: We are also on more than 1,000 different device types.
• #16: With that scale, it is critical to have the right organizational structure to support it.
• #17: Our organization, and therefore our system, is set up to support a distributed architecture.
• #18: I like to think of the Netflix product engineering teams that support development and innovation as being shaped like an hourglass…
• #19: In the top end of the hourglass, we have our device and UI teams who build out great user experiences on Netflix-branded devices. To put that into perspective, there are a few hundred more device types that we support than engineers at Netflix.
• #20: At the bottom end of the hourglass, there are several dozen dependency teams who focus on things like metadata, algorithms, authentication services, A/B test engines, etc.
• #21: The API is at the center of the hourglass, acting as a broker of data.
• #22: This hourglass architecture allows us to scale horizontal easily with our device integrations and our backend dependencies.
• #23: The glue that helps all of this work as effectively as it does is our engineering culture. We hire great, seasoned engineers with excellent judgment and engineering acumen and enable them to build systems quickly by giving them the right context and helping them work together in a highly aligned way.
• #26: With the adoption of the devices, API traffic took off! We went from about 600 million requests per month to about 42 BILLION requests in just two years.
• #27: Fast forward, those numbers have grown to more than 2B incoming requests per day, which translates into more than 14B outbound calls to our dependencies per day.
• #28: Rather than relying on data centers, we have moved everything to the cloud! Enables rapid scaling with relative ease. Adding new servers, in new locations, take minutes. And this is critical when the service needs to grow from 1B requests a month to 2B requests a day in a relatively short period of time.
• #29: That is much more preferable for us than spending our time, money and energy in data centers, adding servers, dealing with power supplies, etc.
• #30: Instead, we spend time in tools such as Asgard, created by Netflix staff, to help us manage our instance types and counts in AWS. Asgard is available in our open source repository at github.
• #31: Another feature afforded to us through AWS to help us scale is Autoscaling. This is the Netflix API request rates over a span of time. The red line represents a potential capacity needed in a data center to ensure that the spikes could be handled without spending a ton more than is needed for the really unlikely scenarios.
• #32: Through autoscaling, instead of buying new servers based on projected spikes in traffic and having systems administrators add them to the farm, the cloud can dynamically and automatically add and remove servers based on need.
• #34: Our distributed architecture, with the number of systems involved, can get quite complicated. Each of these systems talks to a large number of other systems within our architecture.
• #35: Because of this complexity, the API can play a key role in ensuring overall system resiliency. None of our dependency services have SLAs of 100%. Given our unique position in the hourglass as being the last point just before delivery to our users, the API can serve a critical role in protecting our customers from various failures throughout the system.
• #36: In the old world, the system was vulnerable to such failures. For example, if one of our dependency services fails…
• #37: Such a failure could have resulted in an outage in the API.
• #38: And that outage likely would have cascaded to have some kind of substantive impact on the devices.
• #39: The challenge for the API team is to be resilient against dependency outages, to ultimately insulate Netflix customers from low level system problems and to keep them happy.
• #40: To solve this problem, we created Hystrix, as wrapping technology that provides fault tolerance in a distributed environment. Hystrix is also open source and available at our github repository.
• #41: To achieve this, we implemented a series of circuit breakers for each library that we depend on. Each circuit breaker controls the interaction between the API and that dependency. This image is a view of the dependency monitor that allows us to view the health and activity of each dependency. This dashboard is designed to give a real-time view of what is happening with these dependencies (over the last two minutes). We have other dashboards that provide insight into longer-term trends, day-over-day views, etc.
• #42: So, going back to the engineering diagram…
• #43: If that same service fails today…
• #44: We simply disconnect from that service.
• #45: And replace it with an appropriate fallback. The fallback, ideally is a slightly degrade, but useful offering. If we cannot get that, however, we will quickly provide a 5xx response which will help the systems shed load rather than queue things up (which could eventually cause the system as a whole to tip over).
• #46: This will keep ourcustomers happy, even if the experience may be slightly degraded. It is important to note that different dependency libraries have different fallback scenarios. And some are more resilient than others. But the overall sentiment here is accurate at a high level.
• #47: Hystrix and other techniques throughout our engineering organization help keep things resilient. We also have an army of tools that introduce failures to the system which will help us identify problems before they become really big problems.
• #48: The army is the Simian Army, which is a fleet of monkeys who are designed to do a variety of things, in an automated way, in our cloud implementation. Chaos Monkey, for example, periodically terminates AWS instances in production to see how the system as a whole will respond once that server disappears. Latency Monkey introduces latencies and errors into a system to see how it responds. The system is too complex to know how things will respond in various circumstances, so the monkeys expose that information to us in a variety of ways. The monkeys are also available in our open source github repository.
• #50: Going global has a different set of scaling challenges. AWS enables us to add instances in new regions that are closer to our customers.
• #51: To help us manage our traffic across regions, as well as within given regions, we created Zuul. Zuul is open source in our github repository.
• #52: Zuul does a variety of things for us. Zuul fronts our entire streaming application as well as a range of other services within our system.
• #53: Moreover, Zuul is the routing engine that we use for Isthmus, which is designed to marshall traffic between regions, for failover, performance or other reasons.
• #55: Again, we have more than 1,000 different device types that we support. Across those devices, there is a high degree of variability. As a result, we have seen inefficiencies and problems emerge across our implementations. Those issues also translate into issues with the API interaction.
• #56: For example, screen size could significantly affect what the API should deliver to the UI. TVs with bigger screens that can potentially fit more titles and more metadata per title than a mobile phone. Do we need to send all of the extra bits for fields or items that are not needed, requiring the device itself to drop items on the floor? Or can we optimize the deliver of those bits on a per-device basis?
• #57: Different devices have different controlling functions as well. For devices with swipe technologies, such as the iPad, do we need to pre-load a lot of extra titles in case a user swipes the row quickly to see the last of 500 titles in their queue? Or for up-down-left-right controllers, would devices be more optimized by fetching a few items at a time when they are needed? Other devices support voice or hand gestures or pointer technologies. How might those impact the user experience and therefore the metadata needed to support them?
• #58: The technical specs on these devices differ greatly. Some have significant memory space while others do not, impacting how much data can be handled at a given time. Processing power and hard-drive space could also play a role in how the UI performs, in turn potentially influencing the optimal way for fetching content from the API. All of these differences could result in different potential optimizations across these devices.
• #59: Many UI teams needing metadata means many requests to the API team. In the one-size-fits-all API world, we essentially needed to funnel these requests and then prioritize them. That means that some teams would need to wait for API work to be done. It also meant that, because they all shared the same endpoints, we were often adding variations to the endpoints resulting in a more complex system as well as a lot of spaghetti code. Make teams wait due to prioritization was exacerbated by the fact that tasks took longer because the technical debt was increasing, causing time to build and test to increase. Moreover, many of the incoming requests were asking us to do more of the same kinds of customizations. This created a spiral that would be very difficult to break out of…
• #60: That variability ultimately caused us to do some introspection on our API layer.
• #61: Many other companies have seen similar issues and have introduced orchestration layers that enable more flexible interaction models.
• #62: Odata, HYQL, ql.io, rest.li and others are examples of orchestration layers. They address the same problems that we have seen, but we have approached the solution in a very different way.
• #63: We evolved our discussion towards what ultimately became a discussion between resource-based APIs and experience-based APIs.
• #64: The original OSFA API was very resource oriented with granular requests for specific data, delivering specific documents in specific formats.
• #65: The interaction model looked basically like this, with (in this example) the PS3 making many calls across the network to the OSFA API. The API ultimately called back to dependent services to get the corresponding data needed to satisfy the requests.
• #66: In this mode, there is a very clear divide between the Client Code and the Server Code. That divide is the network border.
• #67: And the responsibilities have the same distribution as well. The Client Code handles the rendering of the interface (as well as asking the server for data). The Server Code is responsible of gathering, formatting and delivering the data to the UIs.
• #68: And ultimately, it works. The PS3 interface looks like this and was populated by this interaction model.
• #69: But we believe this is not the optimal way to handle it. In fact, assembling a UI through many resource-based API calls is akin to pointillism paintings. The picture looks great when fully assembled, but it is done by assembling many points put together in the right way.
• #70: We have decided to pursue an experience-based approach instead. Rather than making many API requests to assemble the PS3 home screen, the PS3 will potentially make a single request to a custom, optimized endpoint.
• #71: In an experience-based interaction, the PS3 can potentially make asingle request across the network border to a scripting layer (currently Groovy), in this example to provide the data for the PS3 home screen. The call goes to a very specific, custom endpoint for the PS3 or for a shared UI. The Groovy script then interprets what is needed for the PS3 home screen and triggers a series of calls to the Java API running in the same JVM as the Groovy scripts. The Java API is essentially a series of methods that individually know how to gather the corresponding data from the dependent services. The Java API then returns the data to the Groovy script who then formats and delivers the very specific data back to the PS3.
• #72: We also introduced RxJava into this layer to improve our ability to handle concurrency and callbacks. RxJava is open source in our github repository.
• #73: In this model, the border between Client Code and Server Code is no longer the network border. It is now back on the server. The Groovy is essentially a client adapter written by the client teams.
• #74: And the distribution of work changes as well. The client teams continue to handle UI rendering, but now are also responsible for the formatting and delivery of content. The API team, in terms of the data side of things, is responsible for the data gathering and hand-off to the client adapters. Of course, the API team does many other things, including resiliency, scaling, dependency interactions, etc. This model is essentially a platform for API development.
• #75: If resource-based APIs assemble data like pointillism, experience-based APIs assemble data like a photograph. The experience-based approach captures and delivers it all at once.
• #77: Again, the dependency chains in our system are quite complicated. The API alone includes more than 500 client jars from dependencies when building the application. That complexity makes it virtually impossible to maintain a high degree of nimbleness while having a very high confidence in testing and the code that will be deployed.
• #78: As a result, our philosophy is to act fast (ie. get code into production as quickly as possible), then react fast (ie. response to issues quickly as they arise).
• #79: That said, we do spend a lot of time testing. We just don’t intend to make the system bullet-proof before deploying. Instead, we have employed some techniques to help us learn more about what the new code will look like in production.
• #80: Two such examples are canary deployments and what we call red/black deployments.
• #81: The canary deployments are comparable to canaries in coal mines. We have many servers in production running the current codebase. We will then introduce a single (or perhaps a few) new server(s) into production running new code. Monitoring the canary servers will show what the new code will look like in production.
• #82: If the canary encounters problems, it will register in any number of ways. The problems will be determined based on a comprehensive set of tools that will automatically perform health analysis on the canary.
• #83: If the canary shows errors, we pull it/them down, re-evaluate the new code, debug it, etc.
• #84: We will then repeat the process until the analysis of canary servers look good.
• #85: If the new code looks good in the canary, we can then use a technique that we call red/black deployments to launch the code. Start with red, where production code is running. Fire up a new set of servers (black) equal to the count in red with the new code.
• #86: Then switch the pointer to have external requests point to the black servers. Sometimes, however, we may find an error in the black cluster that was not detected by the canary. For example, some issues can only be seen with full load.
• #87: If a problem is encountered from the black servers, it is easy to rollback quickly by switching the pointer back to red. We will then re-evaluate the new code, debug it, etc.
• #88: Once we have debugged the code, we will put another canary up to evaluate the new changes in production.
• #89: If the new code looks good in the canary, we can then bring up another set of servers with the new code.
• #90: Then we will switch production traffic to the new code.
• #91: If everything still looks good, we disable the red servers and the new code becomes the new red servers.
• #92: All of the open source components discussed here, as well as many others, can be found at the Netflix github repository.