DockerCon2017 - Skypilot
0 likes391 views
This document summarizes the Skypilot project at Demonware to build a containerized continuous delivery pipeline for game services. Key points: - Demonware provides online services for popular games but faced challenges with their legacy monolithic architecture - Skypilot aimed to deliver game services for Skylanders Imaginators through containers and automation to improve agility - It introduced tools like Jenkins, Mesos/Marathon, CoreOS, Docker, and defined workflows for building, configuring and deploying containers - Skylanders launched successfully using this new pipeline, reducing deployment times from hours to minutes and recovery times.
![Building, configuring, testing, deploying [and running]
services
• requires specialist knowledge
• is not safe (incomplete and/or unreliable automation)
• is not efficient
• is not consistent across environments
• does not allow us to work in small batch sizes
The problem](https://image.slidesharecdn.com/dockercon2017-skypilot-170422024327/85/DockerCon2017-Skypilot-9-320.jpg)
DockerCon2017 - Skypilot
- 1. Skypilot Thomas Shaw Build Engineer, Demonware (Activision)
- 2. Delivering Amazing Game Experiences through Containerized Pipelines Skypilot
- 3. 1. Who are Demonware? 2. What is Skypilot? 3. Skypilot Principles 4. Workflows 5. Takeaways Agenda
- 4. Who are Demonware? We provide online services and infrastructure for some of the world’s most popular video game franchises.
- 6. A brief history of Demonware time 2009(MW2) Company size Concurrent users Size of our monolith Number of services Operational overhead Gastric ulcer size 2013 (Ghosts) 2015 (BO3)
- 7. What was Skypilot ? Goal : deliver and run game services for a production title, Skylanders Imaginators. How : through a containerised continuous delivery pipeline. Length : 9 months.
- 8. Deliver an amazing build, test and deploy pipeline that aims to be so reliable, effective and easy to use that our product and title departments will end up writing high value gaming services without giving a second thought to how they may reliably deliver these in record time. Unlocking engineering agility allows us to deliver “amazing value in record time”. Mission Statement
- 9. Building, configuring, testing, deploying [and running] services • requires specialist knowledge • is not safe (incomplete and/or unreliable automation) • is not efficient • is not consistent across environments • does not allow us to work in small batch sizes The problem
- 10. “Evolution of our existing tools was doomed to fail. What we needed was a revolution” - Morgan Brickley, Dev Lead
- 11. This wasn’t a greenfield project. We had some legacy issues and tech debt that we needed to address • Services had always run on bare metal • Monolithic codebase • Internal tooling didn’t support container deployment • Software choices made a decade ago no longer made sense • Processes were inconsistent across teams • Processes were immature Some other challenges
- 12. We want to work in small batch sizes in order to • reduce iteration time • increase quality • fulfil our mandate: unlock engineering agility We also wanted to • codify our deployments • become lean • reduce the entry-level to service deployment The solution
- 13. From hero to zero 1 - Initial 2 - Repeatable 3 - Defined 4 - Managed 5 - Optimizing Process M aturity
- 14. “What’s the smallest number of steps, with the smallest number of people and the smallest amount of ceremony required to get new code running on your servers?” - Erik Kastner
- 16. Keeping it familiar and simple ● git ● docker ● docker-registry ● docker-compose
- 17. Introducing something new ● Jenkins ● Mesos/Marathon ● CoreOS
- 18. Databases ● MySQL in containers ● Data stored on GP2 EBS volumes ● Simple master/slave topology (no master promotion) ● Slaves attempt sync on startup
- 19. Workflows ● Building service containers ● Adding configuration ● Deploying service containers
- 21. Adding Configuration Configuration Shared Build Infrastructure
- 22. Deploying Service Containers Configuration Shared Build Infrastructure Production Infrastructure Restricted Production Artifact Repo
- 23. So how did it work out? Skylanders was successfully launched on October 13th, 2016 without issue.
- 24. Efficiencies Time to build a shared cluster days → 15 - 20 minutes Time to deploy a title environment hours → 10 - 15 minutes Time to recover from a full outage hours → 8 - 15 minutes Time to recover from a database failure hours → 40 - 60 seconds
- 25. What next ? ● Provide a better UX for deployments ● Reduce new deployment time by parallelizing the CD test stage ● Prepare for larger scale deployments ● Educate
- 26. “I was able to update a service from scratch within 45 mins. This included setup time, learning git and 30 mins of user error debugging” - Lisa Reilly, Project Manager
- 27. “With zero Ops experience, I was able to create a Production cluster, on my own, in minutes” - Anar Rahimli, Skypilot customer
- 28. Thank You! @docker #dockercon @demonware