Learning to Scale OpenStack: An Update from the Rackspace Public Cloud

Editor's Notes

• #2: Introductions, welcome to the talk.
• #3: A review of the Rackspace Public Cloud – sets the context for the conversation <number>
• #4: This is our third summit presenting on this topic. Here is a brief review of some of the scale issues we were facing back at the Havana Summit in Portland Our window of time is 30 minutes perceived downtime, 4 hour deploy windows Code coverage wasn't great, lots of errors discovered in production Upstream moved very fast, and we couldn't keep up with all the testing downstream <number>
• #5: Here is a comparison of how we met some of our challenges Our deploys are much faster, some as short as 10 minutes total in our largest environment with 3 minutes of API interruption Deploys are now more reliable Migration data is known ahead of time (and bad ones blocked upstream) We still haven't solved keeping up with upstream. Many factors there. <number>
• #6: We are also learning the countours of openstack, by being the largest public cloud operator. We get to sweat up the hills and coast back down. <number>
• #7: Some of our new challenges – scaling not just deploying bits on nodes as fast as we can. Scaling services Scaling Deployments Scaling Frequency While we are trying to be a thought leader and front runner, collaboration is the key to success. The developer, operator, and testing communities need be aware of these scaling challenges <number>
• #8: Scaling Services – As the size of our cloud grows, and the features of our cloud grows, the services used need to scale along with them. Here we will walk through two scaling scenarios that highlight the challenge. <number>
• #9: Glance is an interesting case. Our glance talks acts as a middle person between HVs and Swift. As glance got used more, the bottleneck emerged. Partly due to our own configuration, but partly due to the nature of glance. Once we resolve the glance issues, swift could be the next bottleneck, care will be needed to make sure we don't just kick performance problems down the line to the next group.
• #10: Nova cells is responsible for interacting between the global cell and all the child cells. Doing this with just a single instance was never going to scale, we just ran out of runway before the pain hit. Through collaboration with upstream, we are now more able to scale out nova-cells as our cell counts grow.
• #11: These challenges will repeat. New bottlenecks will be found and new resource limits will be discovered. Staying ahead of the pain is key. We will not be the only ones to experience this, we are looking for collaboration on how best to manage this kind of scale. <number>
• #12: Our next scale challenge involves deployments. We made great strides around Havana, what have we been doing since? <number>
• #13: Orchestration has been our theme around deployments. We continue to iterate on the parts of the deployment causing the most pain, always making improvements for the next time. Walk through each block and explain why the change was made
• #14: Even with the improvements, we still treat openstack like a legacy application; upgrading in place, not utilizing load balancers, stopping everything to migrate databases, preventing mixed versions, etc.. There are many things that are preventing us from getting to zero downtime, and that's where we can all work together! <number>
• #15: A third scale challenge is frequency. This is the scale of doing things much more often. <number>
• #16: A very relevant quote, but unlike bicycling, when you do something more often in the DevOps world, it does tend to get easier, but there are challenges to going faster! <number>
• #17: Change comes from many sources. These changes need to be distributed to the environments, but with as little customer impact as possible. If we can't deploy changes often enough, we fall behind upstream, we fall behind our features, and we have larger deployments to consume. A snowball effect. Our work on creating new multiple release pipelines, improving our deployment methods, and moving our tests upstream have enabled us to move faster, but not fast enough.
• #18: This is our limit. We absolutely have to make this better. This is a global need, throughout the community of developers, operators, and testers. <number>
• #19: A quick look at what we've got cooking for the Juno cycle <number>
• #20: In Icehouse nova made great strides toward live upgrade with object model and conductor, which give us the ability to run multiple versions of openstack at the same time, notably we could run newer nova-api against an older version in the rest of the environment and shield nova-compute from migrations. This could allow us to roll the update through without downtime of the API and the computes with less interruption. Investigate putting API nodes in read-only during migrations to satisfy some requests and queue others <number>
• #21: This is an ongoing conversation. If we allow each service to work independently, what does that do to the version test matrix? Can we reliably validate anything? While individual projects/services might go faster, does that allow the entire pipeline to go faster? This ties into the discussions happening now at the design summit about cross project interactions. <number>
• #22: Yeah, we need them. Setting them up is hard, lets work together to make them easier. The ops meetups are great for collaborating on the issues at hand. <number>
• #23: We do a lot of things that are hard, but if it wasn't hard, it wouldn't be as satisfying. That's what keeps us coming back. Scaling is more than just tossing code on nodes. There are a lot more considerations to take into account. The development, operator, and tester communities need to collaborate more on where the painful parts are, particularly at scale, and work together on solutions. <number>