Let's build a PaaS platform, how hard could it be?
Download as PPTX, PDF0 likes390 views
The document details the journey of building a PaaS cloud platform, starting from its inception in 2011 with a backend focused on Python and evolving through various challenges and solutions, including application management and performance issues. It highlights the key components like the agent daemon for managing VMs, weight balancing for application resource usage, and eventual adoption of cgroups to contain resource limits. The overall goal was to create an efficient, scalable cloud environment capable of running over 1400 applications with a modernized approach.
Let's build a PaaS platform, how hard could it be?
- 1. Błażej Kasperczyk, Kraków, 05.10.2017 Hey, let's build a PaaS Cloud! ...it's easy, right?
- 3. Team PaaS... 3 • DevOps team • Develop and maintain the Platform • Backend-oriented • Python 3.x, Tornado "With the friends you have in your team, you don't really need enemies!"
- 4. • Approx. 2300 VMs of varying sizes • 1400 active applications, 600 of them in Python3.x • 9000+ running instances • ...a third of it is Python, Tornado-based applications ...And our little cloud – and what runs on it 4
- 5. • Push-button deployment • Scale by available resources and the amount of applications • Quick application installation with our build system • Communications bus between applications The PaaS layer 5
- 6. The slow start • Work started in 2011 • Python2.7 + GEvent • Works over SSH • Push model • Hard limit: 10 applications on each vm • ...and it works! 6
- 7. The inevitable • Approx. 150 VMs max • 300 VMs becomes a hard limit that cannot be bypassed • A single point of failure 7 While the panel was primitive, "papyrus" was a top trending colour!
- 8. • In place of the old orchestrator – a table of states and a coordinator • An API that exposes what needs to be done to reach the desired state • A daemon running on the VM handles the rest • It's 2013 - let's be modern, let's do it in Python3! A moment of reinvention: What if we use our cloud, to scale our cloud? 8
- 9. Scoreboard • Coordinates cloud management • PostgreSQL backend • Responsible for provisioning • Supports over 2000 machines... • ...each querying multiple times every minute... • ...currently. • It can rebuild itself in case of a database failure 9
- 10. Agent daemon • Runs on the VM it manages • Automatically launched with each new VM • Launches and maintains applications • Reports statistics for monitoring purposes • Allows the developer to remotely shut the application down 10
- 11. Density problems • Over-taxing VMs causes performance issues • As it is, the allocation is hit and miss. 11
- 12. Weight balancing • Each VM has a capacity limit • Each application declares its size • Light (White/Green) • Medium (Yellow) • Heavy (Red) • ...that should do it, right? 12
- 13. Oversized cats • A worker can have spikes of 100% CPU usage and 10% averaged. • An application can declare high usage but be harmless. 13
- 14. The RnD • Docker? • LXC? • ...CGroups? 14
- 15. Docker • Requires a major overhaul of our application building and deployment... • ...and will actually do what we already have. 15
- 16. LXC • Current architecture requires a lack of network translation between the Agent and Application... • ...and that caused issues when launching applications 16
- 17. CGroups! • The same mechanism that is used by most containers • Automatic cleanup • Simplicity of the solution 17
- 18. • Applications in the cloud no longer exceed their assigned resources • CPU is limited for each instance • OOMKiller kicks in for memory-heavy applications that tries to exceed its limits Everything is now in a box... 18
- 19. • Time does not stop, or that time we went Xenial and got eaten by SystemD • The Damocles' sword called "Impending Knapsack Problem" • Autoscaling • ...and a few other things ...time to relax, right? 19 As a side effect, we actually made a sane frontend.