AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
Download as PPTX, PDF10 likes4,099 views
The document discusses the use of Apache Mesos for system health analytics, customer experience insights, and anomaly detection in job scheduling and resource management. It highlights the capabilities of various frameworks, like Fenzo and Mantis, for task scheduling, autoscaling, and optimizing resource assignments. Additionally, it includes session titles from a conference on performance tuning, cloud migration, and service reliability.
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
- 12. System health Analytics Customer experience Insights Anomaly detection
- 16. User Job 1 User Job 2 User Job 3 Discovery
- 17. Apache Mesos Apache Mesos Apache Mesos MManantitsis Mantis ASG ASG ASG Fenzo Mesos Framework
- 18. Job Job Job
- 22. FrmWrk1 FrmWrk2 Mesos master Standby master Standby master Mesos slave FrmWrk2 executor TaTasksk Mesos slave FrmWrk1 executor TaTasksk FrmWrk2 executor Mesos slave FrmWrk1 executor TaTasksk
- 24. Instance 1 Instance 1 Instance 2 Task A Task B Instance 1 Task A Instance 1 Task C Task D Task B Task A
- 35. Data stream Task1 Host A Task2 Host B Task3 Host C
- 36. Data stream Data stream Task1 Task2 Task3 Host X Task1 Host A Task2 Host B Task3 Host C
- 41. Mantis Mantis Mantis Fenzo Mesos Framework
- 42. Mesos slave FrmWrk1 executor FrmWrk1 executor Mesos slave Framework executor TaTasksk Task Framework executor Task
- 43. Apache Mesos Apache Mesos Mesos Master Apache Mesos Framework Persistence
- 44. Apache Mesos Apache Mesos Mesos Master Apache Mesos Framework Persistence
- 45. .setName(name) .setFailoverTimeout(to) .setId(id) .setCheckpoint(true) .build();
- 50. Heterogeneous Autoscale Visibility Plugins for Constraints, Fitness High speed
- 52. Mesos master Mesos framework Tasks requests Available resource offers Fenzo task scheduler Persistence
- 53. Fitness Urgency
- 54. Speed Accuracy First fit assignment Optimal assignment Real world trade-offs ~ O (1) ~ O (N * M)1 1 Assuming tasks are not reassigned
- 69. ✔
- 71. 2150 1950 1750 1550 1350 1150 950 750 #Hosts No bin packing used #Full #Partial #Empty 2150 1950 1750 1550 1350 1150 950 750 #Hosts With bin packing #Full #Partial #Empty
- 72. 2050 1950 1850 1750 1650 1550 1450 #Hosts No task runtime-based packer Different runtimes Same runtimes Unused 2050 1950 1850 1750 1650 1550 1450 #Hosts Using task runtime-based packer Different runtimes Same runtimes Unused
- 78. ASG/Cluster: mantisagent MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 ASG/Cluster: mantisagent MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 ASG/cluster: mantisagent MinIdle: 8 MaxIdle: 20 CooldownSecs: 360 Fenzo ScaleUp action: Cluster, N ScaleDown action: Cluster, HostList
- 81. .withLeaseOfferExpirySecs(60) .withLeaseRejectAction( (lease) -> { mesosDriver.declineOffer(lease.getOffer().getId()); })
- 82. .withLeaseOfferExpirySecs(60) .withLeaseRejectAction( (lease) -> { mesosDriver.declineOffer(lease.getOffer().getId()); } ) .withFitnessCalculator( BinPackingFitnessCalculators.cpuBinPacker)
- 85. Heterogeneous Autoscale Visibility Plugins for Constraints, Fitness High speed
- 86. Apache Mesos Apache Mesos Apache Mesos MManantitsis Mantis ASG ASG ASG Fenzo Mesos Framework
- 88. Talk Time Title PFC-305 Wednesday, 1:15pm Embracing Failure: Fault Injection and Service Reliability BDT-403 Wednesday, 2:15pm Next Generation Big Data Platform at Netflix PFC-306 Wednesday, 3:30pm Performance Tuning EC2 DEV-309 Wednesday, 3:30pm From Asgard to Zuul, How Netflix’s proven Open Source Tools can accelerate and scale your services ARC-317 Wednesday, 4:30pm Maintaining a Resilient Front-Door at Massive Scale PFC-304 Wednesday, 4:30pm Effective Inter-process Communications in the Cloud: The Pros and Cons of Micro Services Architectures ENT-209 Wednesday, 4:30pm Cloud Migration, Dev-Ops and Distributed Systems APP-310 Friday, 9:00am Scheduling using Apache Mesos in the Cloud
- 89. Please give us your feedback on this presentation