Gluecon 2013 netflix api crash course
12 likes3,252 views
This document provides an overview of Netflix's API and how it fits into Netflix's systems and operations. The API enables innovation and insulation from failures by personalizing the user experience and enabling mid-tier services and UI scripts. It handles over 2 billion requests per day and is designed for automation, visibility, and balancing speed and quality. The API infrastructure includes load balancers, application clusters, services, and an API layer. Netflix uses tools like distributed tracing, automated canary analysis, deployment automation, and log streaming to operate and monitor the API at scale.
![Distributed grep + tail
2013-05-09.20:38:54 MX 200 us-east-1c i-1824cb73 i-1c61b77f prod
NFPS3-001-8G50FJCX... 288404769389848058 90ms api-global.
netflix.com GET /tvui/release/470/plus/pathEvaluator -
amazon.ami-id: ami-502eb039
amazon.availability-zone: us-east-1c
amazon.instance-id: i-1824cb73
amazon.instance-type: m2.2xlarge
amazon.local-ipv4: 10.6.213.112
amazon.public-hostname: ec2-54-243-4-69.compute-1.amazonaws.
com
amazon.public-ipv4: 54.243.4.69
cookie_esn: NFPS3-001-8G50FJCX...
country: MX
currentTime: 1368131934468
duration-millis: 90
esn: NFPS3-001-8G50FJCX...
geo.city: CIUDADOBREGON
...
$ ./simple_stream.py -f -q 'e["country"]=="MX" && e["esn"]
==~/NFPS3.*/' -r us](https://image.slidesharecdn.com/gluecon2013-netflixapicrashcourse-130524080357-phpapp01/85/Gluecon-2013-netflix-api-crash-course-55-320.jpg)
Gluecon 2013 netflix api crash course
- 1. Netflix API Crash Course Building & Running the API in 30 minutes Ben Schmaus, Netflix May 2013, Gluecon bschmaus@netflix.com @schmaus
- 2. Streaming TV Shows & Movies Globally
- 6. More than 36 million subscribers in over 40 countries
- 7. How does the API fit into the picture?
- 9. Personalization Engine User Info Movie Metadata Ratings Similar Movies Instant Queue A/B Test Engine API Enable UX Innovation Insulate from Failure
- 10. > 2 Billion Requests per Day
- 11. Growth Over Time
- 24. How's the API put together?
- 27. Inside an API App Server RxJava Hystrix Service Client 1 Service Client 2 Service Client N
- 28. HystrixRx+Java Service Layer Service Client (provided JAR) Application Service /device/endpoint (provided script) Service UI Teams Mid-tier Service Teams API Team
- 29. Continually changing UI scripts and mid-tier services Functionality, resiliency and performance drifts over time
- 30. Deployment & Ops
- 31. REMOVE MANUAL WORK pushing code to multiple AWS regions/clusters ENABLE RAPID DEPLOYMENT of code despite limited visibility into how it's changed KEEP TEAM INFORMED about what's happening in prod MITIGATE RISK of systemic failure
- 32. Tools
- 34. Code Flow
- 35. Run 1% of your traffic on the new code and see how it does
- 36. API ami-123 API ami-456 2xx 4xx 5xx latency busy threads load ...
- 37. Manually looking at graphs and SSH- ing into servers and grep-ing logs doesn't scale (although we used to do that)
- 38. Confidence score for each AMI based on comparison of 1000+ metrics
- 39. Scannable visualization of metric space More important Less important
- 40. Cross-reference Jira, Link to code diffs
- 42. Easy to access report artifacts for each AMI
- 43. Your basic red/black push
- 48. Doing red/black by hand for multiple clusters across multiple regions is not fun
- 50. Automate multi-cluster/region pushes Don't forget to automate rollbacks, too!
- 51. $Who, $What, $Where, $When e.g., "bschmaus, ami-123, Sandbox Canary, 2013-05-06 19:05" Latest prod change in chat topic
- 52. Quickly see status of all clusters in a region
- 53. What the #%*! just happened!?
- 54. Historical & realtime metrics, sort realtime by error/request rate
- 55. Distributed grep + tail 2013-05-09.20:38:54 MX 200 us-east-1c i-1824cb73 i-1c61b77f prod NFPS3-001-8G50FJCX... 288404769389848058 90ms api-global. netflix.com GET /tvui/release/470/plus/pathEvaluator - amazon.ami-id: ami-502eb039 amazon.availability-zone: us-east-1c amazon.instance-id: i-1824cb73 amazon.instance-type: m2.2xlarge amazon.local-ipv4: 10.6.213.112 amazon.public-hostname: ec2-54-243-4-69.compute-1.amazonaws. com amazon.public-ipv4: 54.243.4.69 cookie_esn: NFPS3-001-8G50FJCX... country: MX currentTime: 1368131934468 duration-millis: 90 esn: NFPS3-001-8G50FJCX... geo.city: CIUDADOBREGON ... $ ./simple_stream.py -f -q 'e["country"]=="MX" && e["esn"] ==~/NFPS3.*/' -r us
- 56. Go for haystack handing you the needle
- 57. Or at least be able to make smaller haystacks
- 58. Continuously experiment to make hard things easier
- 59. Even with the best tools, building software is hard work. Great engineers build great software.
- 60. Want to help us build the API? bschmaus@netflix.com @schmaus