Scaling infrastructure beyond containers
1 like798 views
The document outlines a presentation on scaling infrastructure beyond containers, focusing on the evolution of infrastructure at Allegro and the adoption of Apache Mesos. Key challenges discussed include effective resource utilization, cloud computing limitations, and the need for performance isolation in microservice architectures. The future of data centers is also explored, highlighting the importance of fine-grained resource sharing and advanced scheduling capabilities.
Scaling infrastructure beyond containers
- 1. @wendigo Mateusz „Serafin” Gajewski • AWS UG Meetup Scaling infrastructure beyond containers
- 2. @wendigo Agenda • Evolution of infrastructure at Allegro, • Why Apache Mesos™? • Apache Mesos key concepts, • Future of datacenter and cloud computing?
- 3. @wendigo History of scaling infrastructure @ Allegro
- 6. @wendigo Web Scale Resource management 100s dots 100s dots
- 7. @wendigo Infrastructure 2.0 1000s dots another 1000s dots
- 8. @wendigo Infrastructure 2.1 1000s dots another 1000s dots
- 9. @wendigo Challenges • cloud not used as cloud ;), • high cost of virtualization, • effective resource utilization, • microservice architecture, • spread of new technologies, • heterogenous resources, • scalability, fault tolerance & HA, • performance isolation, • data processing at scale
- 11. @wendigo Holly Grail of TCO
- 12. @wendigo Infrastructure 3.0 A Platform for Fine-Grained Resource Sharing in the Data Center
- 17. @wendigo Offers
- 20. @wendigo Mesos HA • master election/failover with ZooKeeper, • master maintains soft-state, • framework state reconciliation, • slave checkpointing, • slave recovery, • framework checkpointing
- 21. @wendigo Beyond offers • offer filters (constraints), • static (pre-startup) reservations, • dynamic (post-startup) reservations, • oversubscription, • persistent volumes, • pluggable allocator scheduling policy (fair, priority based)
- 24. @wendigo Mesos recap • „programming against the datacenter", • distributed datacenter kernel, • two-level multi-resource scheduler, • scalable, highly-available & fault-tolerant, • performance isolation with containers, • exposes homogeneous resources, • elastic, dynamic partitioning, • high resource utilization
- 26. @wendigo IaaC
- 28. @wendigo Google’s Omega source: Omega: flexible, scalable schedulers for large compute clusters
- 30. Questions?