Prezo at-mesos con2015-final
3 likes1,777 views
The document discusses the use of Apache Mesos for heterogeneous resource scheduling in cloud-native frameworks, focusing on the Fenzo scheduler library utilized by Netflix. It highlights Fenzo's capabilities, including resource granularity, task placement constraints, and cluster autoscaling strategies, while addressing the challenges of developing a new framework for handling diverse workloads. The document also provides insights into the library's performance metrics, scheduling optimizations, and future directions for enhancing task management and integrating new Mesos features.
Prezo at-mesos con2015-final
- 1. Heterogeneous Resource Scheduling Using Apache Mesos for Cloud Native Frameworks Sharma Podila Senior Software Engineer Netflix Aug 20th MesosCon 2015
- 2. Agenda ● Context, motivation ● Fenzo scheduler library ● Usage at Netflix ● Future direction
- 3. Why use Apache Mesos in a cloud? Resource granularity Application start latency
- 4. A tale of two frameworks Reactive stream processing Container deployment and management
- 5. Reactive stream processing, Mantis ● Cloud native ● Lightweight, dynamic jobs ○ Stateful, multi-stage ○ Real time, anomaly detection, etc. ● Task placement constraints ○ Cloud constructs ○ Resource utilization ● Service and batch style
- 6. Mantis job topology Worker SourceApp Stage1 Stage2 Stage3 Sink A job is set of 1 or more stages A stage is a set of 1 or more workers A worker is a Mesos task Job
- 7. Container management, Titan ● Cloud native ● Service and batch workloads ● Jobs with multiple sets of container tasks ● Container placement constraints ○ Cloud constructs ○ Resource affinity ○ Task locality
- 8. Job 2 Set 0 Set 1 Container scheduling model Job 1 Co-locate tasks from multiple task sets
- 9. Why develop a new framework?
- 10. Easy to write a new framework?
- 11. Easy to write a new framework? What about scale? Performance? Fault tolerance? Availability?
- 12. Easy to write a new framework? What about scale? Performance? Fault tolerance? Availability? And scheduling is a hard problem to solve
- 13. Long term justification is needed to create a new Mesos framework
- 14. Our motivations for new framework ● Cloud native (autoscaling)
- 15. Our motivations for new framework ● Cloud native (autoscaling) ● Customizable task placement optimizations (Mix of service, batch, and stream topologies)
- 16. Cluster autoscaling challenge Host 1 Host 2 Host 3 Host 4 Host 1 Host 2 Host 3 Host 4 vs. For long running stateful services
- 17. Cluster autoscaling challenge Host 1 Host 2 Host 3 Host 4 Host 1 Host 2 Host 3 Host 4 vs. For long running stateful services
- 18. Components of a mesos framework API for users to interact
- 19. Components of a mesos framework API for users to interact Be connected to Mesos via the driver
- 20. Components of a mesos framework API for users to interact Be connected to Mesos via the driver Compute resource assignments for tasks
- 21. Components of a mesos framework API for users to interact Be connected to Mesos via the driver Compute resource assignments for tasks
- 22. Fenzo A common scheduling library for Mesos frameworks
- 23. Fenzo usage in frameworks Mesos master Mesos framework Tasks requests Available resource offers Fenzo task scheduler Task assignment result • Host1 • Task1 • Task2 • Host2 • Task3 • Task4 Persistence
- 24. Fenzo scheduling library Heterogeneous resources Autoscaling of cluster Visibility of scheduler actions Plugins for Constraints, Fitness High speed Heterogeneous task requests
- 25. Announcing availability of Fenzo in Netflix OSS suite
- 26. Fenzo details
- 27. Scheduling problem Fitness Pending Assigned Urgency N tasks to assign from M possible slaves
- 28. Scheduling optimizations Speed Accuracy First fit assignment Optimal assignment Real world trade-offs~ O (1) ~ O (N * M)1 1 Assuming tasks are not reassigned
- 29. Scheduling strategy For each task On each host Validate hard constraints Eval fitness and soft constraints Until fitness good enough, and A minimum #hosts evaluated
- 33. Host attribute value constraint Task HostAttrConstraint:instanceType=r3 Host1 Attr:instanceType=m3 Host2 Attr:instanceType=r3 Host3 Attr:instanceType=c3 Fenzo
- 34. Unique host attribute constraint Task UniqueAttr:zone Host1 Attr:zone=1a Host2 Attr:zone=1a Host3 Attr:zone=1b Fenzo
- 35. Balance host attribute constraint Host1 Attr:zone=1a Host2 Attr:zone=1b Host3 Attr:zone=1c Job with 9 tasks, BalanceAttr:zone Fenzo
- 36. Fitness evaluation Degree of fitness Composable
- 37. Bin packing fitness calculator Fitness for Host1 Host2 Host3 Host4 Host5 fitness = usedCPUs / totalCPUs
- 38. Bin packing fitness calculator Fitness for 0.25 0.5 0.75 1.0 0.0 fitness = usedCPUs / totalCPUs Host1 Host2 Host3 Host4 Host5
- 39. Bin packing fitness calculator Fitness for 0.25 0.5 0.75 1.0 0.0 ✔ Host1 Host2 Host3 Host4 Host5 fitness = usedCPUs / totalCPUs Host1 Host2 Host3 Host4 Host5
- 40. Composable fitness calculators Fitness = ( BinPackFitness * BinPackWeight + RuntimePackFitness * RuntimeWeight ) / 2.0
- 41. Cluster autoscaling in Fenzo ASG/Cluster: mantisagent MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 ASG/Cluster: mantisagent MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 ASG/cluster: computeCluster MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 Fenzo ScaleUp action: Cluster, N ScaleDown action: Cluster, HostList
- 42. Rules based cluster autoscaling ● Set up rules per host attribute value ○ E.g., one autoscale rule per ASG/cluster, one cluster for network- intensive jobs, another for CPU/memory-intensive jobs ● Sample: #Idle hosts Trigger down scale Trigger up scale min max Cluster Name Min Idle Count Max Idle Count Cooldown Secs NetworkClstr 5 15 360 ComputeClstr 10 20 300
- 43. Shortfall analysis based autoscaling ● Rule-based scale up has a cool down period ○ What if there’s a surge of incoming requests? ● Pending requests trigger shortfall analysis ○ Scale up happens regardless of cool down period ○ Remembers which tasks have already been covered
- 44. Usage at Netflix
- 46. Scheduler run time (milliseconds) Scheduler run time in milliseconds (over a week) Average 2 mS Maximum 38 mS Note: times can vary depending on # of tasks, # and types of constraints, and # of hosts
- 47. Experimenting with Fenzo Note: Experiments can be run without requiring a physical cluster
- 48. A bin packing experiment Host 3 Host 2 Host 1 Mesos Slaves Host 3000 Tasks with cpu=1 Tasks with cpu=3 Tasks with cpu=6 Fenzo Iteratively assign Bunch of tasks Start with idle cluster
- 49. Bin packing sample results Bin pack tasks using Fenzo’s built-in CPU bin packer
- 50. Bin packing sample results Bin pack tasks using Fenzo’s built-in CPU bin packer
- 51. Task runtime bin packing sample Bin pack tasks based on custom fitness calculator to pack short vs. long run time jobs separately
- 52. Scheduler speed experiment # of hosts # of tasks to assign each time Avg time Avg time per task Min time Max time Total time Hosts: 8-CPUs each Task mix: 20% running 1-CPU jobs, 40% running 4-CPU, and 40% running 6-CPU jobs Goal: starting from an empty cluster, assign tasks to fill all hosts Scheduling strategy: CPU bin packing
- 53. Scheduler speed experiment # of hosts # of tasks to assign each time Avg time Avg time per task Min time Max time Total time 1,000 1 3 mS 3 mS 1 mS 188 mS 9 s 1,000 200 40 mS 0.2 mS 17 mS 100 mS 0.5 s Hosts: 8-CPUs each Task mix: 20% running 1-CPU jobs, 40% running 4-CPU, and 40% running 6-CPU jobs Goal: starting from an empty cluster, assign tasks to fill all hosts Scheduling strategy: CPU bin packing
- 54. Scheduler speed experiment Hosts: 8-CPUs each Task mix: 20% running 1-CPU jobs, 40% running 4-CPU, and 40% running 6-CPU jobs Goal: starting from an empty cluster, assign tasks to fill all hosts Scheduling strategy: CPU bin packing # of hosts # of tasks to assign each time Avg time Avg time per task Min time Max time Total time 1,000 1 3 mS 3 mS 1 mS 188 mS 9 s 1,000 200 40 mS 0.2 mS 17 mS 100 mS 0.5 s 10,000 1 29 mS 29 mS 10 mS 240 mS 870 s 10,000 200 132 mS 0.66 mS 22 mS 434 mS 19 s
- 55. Accessing Fenzo Code at https://github.com/Netflix/Fenzo Wiki at https://github.com/Netflix/Fenzo/wiki
- 56. Future directions ● Task management SLAs ● Support for newer Mesos features ● Collaboration
- 57. To summarize...
- 58. Fenzo: scheduling library for frameworks Heterogeneous resources Autoscaling of cluster Visibility of scheduler actions Plugins for Constraints, Fitness High speed Heterogeneous task requests
- 59. Fenzo is now available in Netflix OSS suite at https://github.com/Netflix/Fenzo
- 60. Questions? Heterogeneous Resource Scheduling Using Apache Mesos for Cloud Native Frameworks Sharma Podila spodila @ netflix . com @podila