Growing OpenStack at TWC, 3rd Speaker

Editor's Notes

• #2: Simplified slide will amplify quite a bit while speaking: Mailing lists for each team, mailing list for deployment strategy (now a distinc openstack team) Meetings: IRC meetings for each project. IRC meetings for deployment tools. IRC or similar Meetings for “not quite” a project (like Monasca). F2F meetings including summits, design mid-cycle and sprints. F2F midcycle operators meetup The groups.openstack.org to find a local community as well as the larger community Community processes for bug filing, but much more around revieiwng & tagging blueprints & processes around influencing openstack with contributions, voting, etc. Just Be Involved. Be familiar with the entire end to end bug->patch->review->commit->backport-to-stable process as you will care and when you care, you may not have time to familiarize yourself. Lots of tools including git review, gerrit, jenkins usage, launchpad, the blueprinting tools, etc. And the tools ever grow in scope and variety, so stay abreast of changes. In 18 months, everything you know about OpenStack is obsolete.Grenade, Rally, DevStack,etc. Key point: There’s value in the discussions that take place each day in IRC, F2F, mailing lists, etc. There is value in what’s out there. There is value in YOU, in YOUR EXPERIENCES, you bring value into openstack and you need to be a community participant to share that value with others.
• #4: RabbitMQ Message broker used by almost all openstack components to communicate with each other Rabbit failures are difficult to detect and have wide effects May manifest itself, for example volume attach randomly failing It takes time to determine a pattern of behavior for these types of failures and then figure out that rabbit is having issues. Pay attention to RabbitMQ Queues (make sure there aren’t messages in Ready State or unACKed.) This is a focus area for us and the operators community Adding monitoring Adding some automated queue cleanup Heartbeat support Focus Easier to use More reliable
• #5: Neutron (OpenStack Networking) When neutron has problems, guest VMs can lose network access and customers get upset OpenvSwitch crashes, newer is usually better don’t be the first one to try a new feature (Distributed Virtual Router or HA routers) have maintenance/migration plans in place created ansible tools for this
• #6: Kernel panics we’ve had issues in the past with kernel panics on compute nodes (hypervisors) and control nodes. Wev’e had a CEPH node kernel panic. We’ve upgraded the kernel several times and hope to have mitigated this some. You need a plan to handle how you will do maintenance like kernel upgrades Kernel debugging is not our strong suit, nor an area we intend to grow, so we relied heavily on Canonical to provide analysis and kernel fixes.
• #7: Users Users present their own challenges Our best response to these challenges has been training and documentation. OpenStack, and more broadly cloud computing, represents a cultural shift for some users who may be used to running on dedicated (and expensive) five 9s hardware, their app may not be cloud-ready Reinforce this message regularly Build tooling to support users. you need an on-board process to setup accounts, projects, etc. You need a similar process to offboard when people leave the company, you might need to plan what happens to a VM that a whole team is using that was created by a guy who just resigned, who takes ownership of the resources? you need a ticketing system and you might need at least an informal SLA with your users Pleasant Surprises are also Pain Points as users may adopt (or demand) newer features than you have deployed, or features you have deployed but have not yet internally trained on Heat Anti-affinity Affinity Develop test cases around these areas of user uptake.
• #9: Better integration testing Each “event” results in more testing, each new service results in more testing, and each “time of continued service” represents opportunities for more system level testing (performance, etc) We have tools and techniques that still need automation and integration. We use both Ansible and Puppet and some of each have “border” cases wherein we could be doing better integration. Ie., RGE, ansible_adhoc, etc. We have opportunities to do more automation to get us to daily/weekly curated releases of individual OpenStack services We want to move away from distro packaging to more upstream-like python virtual environments to enable better continuous deployment
• #10: Designate, DNSaaS has just been released into our release train LBaaS is in our staging and devel environment Monitoring as a service (Monasca) is used in our infrastructure but will also be provided as a service to our users for their VMs DBaaS is coming soon Hadoop (Sahara) is coming soon
• #12: Augmented with partners SwiftStack Inktank Cisco Had to mobilize other teams in TWC Network Engineering Network Ops Data Center Services Etc.