Netflix container scheduling talk at stanford final
0 likes625 views
The document discusses Netflix's approach to elastic resource scheduling within its scalable container cloud infrastructure, specifically using Apache Mesos and Fenzo for task scheduling and resource management. It highlights the benefits of transitioning from virtual machines to containers, including faster deployment and improved development efficiency, as well as various scheduling optimizations and strategies employed to handle Netflix's diverse workloads. The document also addresses future plans for enhancing scheduling capabilities, autoscaling, and optimizing resource allocation.
Netflix container scheduling talk at stanford final
- 1. Elastic resource scheduling for Netflix's scalable container cloud Sharma Podila, Andrew Spyker, Tomasz Bak Feb 7th 2017
- 2. Topics â Motivations for containers on AWS EC2 â Scheduling using Apache Mesos â Fenzo deep dive â Future plans
- 3. Containers add to our VM infrastructure Already in VMâs have ... microservice driven, cloud native, CI/CD devops enabled, resilient, elastically scalable, environment
- 4. Container Provides Innovation Velocity â Iterative local development, deploy when ready â Manage app and dependencies easily and completely â Simpler way to express resources, let system manage
- 5. Service Batch Sampling of container usage Media Encoding Digital Watermarking NodeJS UI Services Operations and General Stream Processing Reporting
- 6. Sampling of realized container benefits â Media Encoding - encoding research development time â VMâs platform to container platform - 1 month vs. 1 week â Continuous Integration Testing â Build all Netflix codebases in hours â Saves development 100âs of hours of debugging â Netflix API Re-architecture using NodeJS â Focus returns to app development â Provided reliable smaller instances â Simplifies, speeds test and deployment
- 8. Reactive stream processing: Mantis Zuul Cluster API Cluster Mantis Stream processing Cloud native service â Configurable message delivery guarantees â Heterogeneous workloads â Real-time dashboarding, alerting â Anomaly detection, metric generation â Interactive exploration of streaming data Anomaly Detection
- 9. Current Mantis usage â At peak: â 2,300 EC2 instances of M3.2xlarge instances â Peak of 900 concurrent jobs â Peak of 5,200 concurrent containers â Trough of 4,000 containers â Job sizes range from 1 to 500 containers â Mix of perpetual and interactive exploratory jobs â Peak of 13 Million events / sec
- 10. EC2 VPC VMVM TitusJob Control Containers App Cloud Platform (metrics, IPC, health) VMVM Batch Containers Eureka Edda Container deployment: Titus Atlas & Insight
- 11. Current Titus usage #Containers (tasks) for the week of 11/7 in one of the regions â Peak of ~1,800 instances â Mix of m4.4xl, r3.8xl, p2.8xl â ~800 instances at trough â Mix of batch, stream processing, and some microservices
- 12. Core architectural components AWS EC2 Apache Mesos Titus/Mantis Framework Fenzo Fenzo at https://github.com/Netflix/Fenzo Apache Mesos at http://mesos.apache.org/ Batch Job Mgr Service Job Mgr
- 13. Scheduling using Apache Mesos
- 15. Motivation for a new Mesos scheduler â Cloud native (cluster autoscaling) â Customizable task placement optimizations â Mix of service, batch, and stream topologies
- 16. What does a Mesos scheduler do? â API for users to interact â Mesos interaction via the driver â Compute resource assignments for tasks
- 17. What does a Mesos scheduler do? â API for users to interact â Be connected to Mesos via the driver â Compute resource assignments for tasks â NetflixOSS Fenzo https://github.com/Netflix/Fenzo
- 18. Fenzo deep dive
- 19. Scheduling optimizations Speed Accuracy First fit assignment Optimal assignment Real world trade-offs
- 20. Fitness Pending Assigned Urgency Scheduling problem N tasks to assign from M possible agents
- 22. DC/Cloud operator Scheduling optimizations â Bin packing â By resource usage â By job types â Ease deployment of new agent AMIs â Ease server maintenance and upgrades
- 23. DC/Cloud operator Application owner â Task locality, anti-locality (noisy neighbors?, etc.) â Resource affinity â Task balancing across racks/AZs/hosts Scheduling optimizations
- 24. DC/Cloud operator Application owner Cost â Save cloud footprint costs â Right instance types â Save power, cooling costs â Does everything need to run right away? Scheduling optimizations
- 25. DC/Cloud operator Application owner Cost Security Security aspects of multi-tenant applications on a host Scheduling optimizations
- 26. DC/Cloud operator Application owner Cost Security Proceed quickly in the generally right direction, adapting to changes Scheduling optimizations
- 27. â Extensible â Cloud native â Ease of experimentation â Scheduling decisions visibility Fenzo goals
- 28. Fenzo scheduling strategy For each (ordered) task On each available host Validate hard constraints Score fitness and soft constraints Until score good enough, and A minimum #hosts evaluated Pick host with highest score
- 29. Experimentation with Fenzo â Abstractions of tasks and servers (VMs) â Create various strategies with custom fitness functions and constraints â For example, dynamic task anti-locality â âGood enoughâ can be dynamic â Based on pending task set size, task type, etc. â Ordering of servers for allocation based on task type
- 30. Experimentation with Fenzo Task runtime bin packing sample results Resource bin packing sample results
- 31. Fitness function Vs. constraints â Fitness: site policies â Bin packing for utilization, reduce fragmentation â Segregate hosts by task types, e.g., service Vs batch â Constraints: user preferences â Resource affinity â Task locality â Balance tasks across racks or availability zones
- 32. â Degree of fitness, score of 0.0 - 1.0 â Composable â Multiple weighted fitness functions â Extensible â Combine existing ones with custom plugins Fitness evaluation
- 33. CPU bin packing fitness function Fitness for Host1 Host2 Host3 Host4 Host5 fitness = usedCPUs / totalCPUs Host1 Host2 Host3 Host4 Host5
- 34. Fitness for 0.25 0.5 0.75 1.0 0.0 Host1 Host2 Host3 Host4 Host5 fitness = usedCPUs / totalCPUs Host1 Host2 Host3 Host4 Host5 CPU bin packing fitness function
- 35. Fitness for 0.25 0.5 0.75 1.0 0.0 â Host1 Host2 Host3 Host4 Host5 fitness = usedCPUs / totalCPUs Host1 Host2 Host3 Host4 Host5 CPU bin packing fitness function
- 36. Combines resource request bin packing with task type bin packing resBinpack = (cpuFit + memFit + networkFit) / 3.0 taskTypePack = numSameType / totTasks fitness = resBinpack * 0.4 + taskTypePack * 0.6 Current fitness evaluator in Titus
- 37. Fenzo constraints â Common constraints built-in â Host attribute value â Host with unique attribute value â Balance across hostsâ unique attribute value â Can be used as âsoftâ or âhardâ constraint â Soft evaluates to 0.0 - 1.0 â Hard evaluates to true/false â Additional custom plugins â Global constraint to send only GPU requiring tasks to GPU hosts â Global constraint to limit EC2 instance types to certain tasks
- 38. â CPU â Memory â Disk â Ports â Network bandwidth â Scalar (used for GPU) â Security groups and IP per container Fenzo supported resources
- 39. Why is a task failing to launch?
- 40. Fenzo cluster autoscaling Host 1 Host 2 Host 3 Host 4
- 41. Fenzo cluster autoscaling Host 4Host 3Host 1 vs. Host 1 Host 2 Host 2 Host 3 Host 4
- 42. Fenzo cluster autoscaling Host 4Host 3Host 1 vs. Host 1 Host 2 Host 2 Host 3 Host 4
- 43. Fenzo cluster autoscaling â Threshold based â Shortfall analysis based Host 4Host 3Host 1 vs. Host 1 Host 2 Host 2 Host 3 Host 4
- 44. Autoscaling multiple agent clusters m4.4xlarge agents r3.8xlarge agents Titus Grouping agents by instance type letâs us autoscale them independently Min Desired Max Min Desired Max
- 45. Threshold based autoscaling â Set up rules per agent attribute value â Sample: #Idle hosts Trigger down scale Trigger up scale min max Cluster Name Min Idle Max Idle Cooldown Secs MemosyClstr 2 5 360 ComputeClstr 5 10 300
- 46. Shortfall analysis based scale up â Rule-based scale up has a cool down period â What if thereâs a surge of incoming requests? â Pending requests trigger shortfall analysis â Scale up happens regardless of cool down period â Remembers which tasks have already been covered â Shortcoming: scale can be too aggressive for short periods of time
- 48. â Guarantee capacity for timely job starts â Mesos supports quotas, but, inadequate at this time â Generally, optimize throughput for batch jobs and start latency for service jobs â Categorize by expected behavior â For example, some service style jobs may be less important â Critical versus Flex (flexible) scheduling requirements Capacity guarantees
- 51. Capacity guarantees: hybrid view AppC1 AppC2 AppC3 AppCN AppF1 AppF2 AppFN AppF3 Resource Allocation Order Critical Flex
- 52. Capacity guarantees: hybrid view Tier Capacity = SUM (App1-cap + App2-cap + ⊠+ AppN-cap) + BUFFER BUFFER: â Accommodate some new or ad hoc jobs with no guarantees â Red-black pushes of apps temporarily double app capacity AppC1 AppC2 AppC3 AppCN AppF1 AppF2 AppFN AppF3 Resource Allocation Order Critical Flex
- 53. Capacity guarantees: hybrid view Fenzo supports multi-tiered task queues Can have arbitrary number of tiers Per tier DRF across multiple queues Tier 0 Tier 1
- 54. Sizing clusters for capacity guarantees Tier 0: Used capacity Idle capacity Autoscaled Cluster min size (guaranteed capacity) Cluster max Size Tier 1: Used capacity Autoscaled Cluster desired size Cluster max Size (Idle size kept near zero)
- 55. Netflix container execution values
- 56. Netflix container execution values â Consistent cloud infrastructure with VMâs â Virtualize and deeply re-use AWS features â User and operator tooling common to VMâs â IPC and service discovery, telemetry and monitoring â Spinnaker integration for CI/CD â Unique Features â Deep Amazon and Netflix infrastructure integration â VPC IP per container â Advanced security (sec groups, IAM Roles)
- 57. Elastic Network Interfaces (ENI) AWS EC2 Instance ENI0 IP0 IP1 IP2 IP3 ENI1 IP4 IP5 IP6 IP7 ENI2 IP8 IP9 IP10 IP11 ENI0 IP0 IP1 IP2 IP3 â Each EC2 instance in VPC has 2 or more ENIs â Each ENI can have 2 or more IPs â Security Groups are set on the ENI
- 58. Network bandwidth isolation Each container gets an IP on one of the ENIs Linux tc policies used on virtual Ethernet For both incoming and outgoing traffic Bandwidth limited to the requested value No bursting into unused bandwidth
- 59. GPU Enablement Personalization and recommendations â Deep learning with neural nets/mini batch â Makes model training infrastructure self-service Executor takes Scheduler resource definition â Maps p2.8xl GPUâs using nvidia-docker-plugin â Mounts drivers and devices into container
- 60. â Fine grain capacity guarantees â DRF adapted to elastic clusters â Preemptions to improve resource usage efficiency â Hierarchical sharing policies via h-DRF â Leveraging âInternal spot marketâ, aka the trough â Onboarding new applications â Scale continues to grow Ongoing, and future scheduling work
- 61. Elastic resource scheduling for Netflix's scalable container cloud Sharma Podila, Andrew Spyker, Tomasz Bak @podila @aspyker @tomaszbak1974 Questions?