Delivering a bleeding edge community-led openstack distribution: RDO
- 1. Delivering a bleeding edge community-led OpenStack distribution: RDO Chandan Kumar at FOSSASIA 2018
- 2. About me: ● RDO/OpenStack Contributor ● Software Engineer at Red Hat
- 3. About RDO ● The rpm based distribution of upstream OpenStack ● But also a group of Ridiculously Dedicated OpenStackers.. ● Who are here to help you Rapidly Deploy OpenStack ● 350 + packages maintained, and growing
- 4. What’s in a package? ● <Name>-<Version>-<Release>.rpm ● Source + spec file = package ● Source -> upstream code ● List of included files ● What is in a spec file: ○ Building Steps ○ Installation/Removal steps ○ Dependencies ○ Patches
- 5. Packager == Spec Developer ● Version control (dist-git), on a public platform, community oriented ● Peer review ● Automated testing & validation of contributions (ie continuous integration), on a controlled testing environment ● Smart merge management ● Responsible with system integration like: systemd unit or specific path etc. ● Follow upstream changes in order to validate spec files against them ● Release as closely as possible to upstream (esp. vulnerability or security fixes)
- 7. Current State till Queens release. ● 700+ projects ● 1600 + contributors ● 250 commits per day ● Releases every 6 months
- 8. Extra constraints for RDO Packagers ● Spec file validation on each commit merged ● Strong dependencies management between packages ● Test system should properly test and validate the package. ● Package upgrade should works smoothly. ● Making sure Package works on Fedora, Centos/RHEL
- 9. RDO releases status Release Upstream OpenStack RDO releases Duration (in days) Newton 06th Oct, 2016 07th Oct, 2017 1 Ocata 22nd Feb, 2017 22nd Feb, 2017 In 12 hours Pike 30th Aug, 2017 01st Sept, 2017 In 2 days * Queens 01st Mar, 2018 01st Mar, 2018 In 7 hours * Production push ready on 28th Aug, 2017, Infra issues leads few days delay.
- 10. RDO got released on the same day as upstream released? But how it got possible?
- 11. How we made it possible? ● Building RPM Factory from Software Factory, RDO and community tools ● Started chasing the trunk ● Testing each commit as it merged upstream through puppet and Tripleo CI ● More scenario jobs leads to more robust openstack ● Catching issues early once a patch landed upstream through RDO third party CI. ● Improving RDO Production chain workflow
- 12. Community Tools: Koji and Centos Cloud Sig ● Koji is a software collection to build and store RPM packages in an isolated environment. ● Centos Build System: http://cbs.centos.org/koji/ ● Centos Cloud Sig providing Centos Infra and CI to ship packages for stable releases. ● CICO <ci.centos.org> cloud for providing additional CI capacity. ● Dogfooding our own openstack through RDO Cloud
- 13. RDO Tools: DLRN ● Continuous Delivery Platform that takes every upstream commit and build it as a consumable package for CI purpose New commit current Consistent Current Passed CI RDO CI Upstream CI (Tripleo, Kolla, Puppet)every merged commit or almost is built. set of packages without build failures snapshot that passed CI
- 14. RDO Tools: rdopkg, rdoinfo, config and graffiti ● RPM packaging automation tool providing more power to packagers ● Automates most of the packaging tasks like requirements sync or flattening patch chains for inclusion in specs ● rdoinfo is a git repo containing all the information about openstack packages and its dependencies including maintainers, version, dist-git, upstream and a lot more. ● Graffiti is a utility tool in order to interact with koji to retrieve releases and tags ● Config: Treating each config changes for jobs and infra as code using git
- 15. RDO originated tools: WeiRDO ● Ansible-based framework using gate jobs and make them usable outside upstream gates ● Goal is to improve coverage upon what upstream provides. provision test node Setup trunk repositor ies Install project test dependen cies execute run_test. sh Deploy/c onfigure opensta ck Run tempest tests recover logs and results destroy node
- 16. RDO originated tools: ARA ● Non-intrusive plugin to Ansible. ● Tool to records Ansible Playbook runs and provides intuitive interfaces to browse them. ● Used heavily in upstream CI and openstack-ansible project ● Look for ‘ara’ folder in CI logs ● Making it easier to end-user to trace errors ● Try out through $ pip install ara
- 17. Software Factory ● SF is a CI/CD platform based on OpenStack's own CI ("Infra") ● Code hosting and review (Gerrit) ● Jobs orchestration (Zuul & Jenkins) ● Project dependencies management when building testing environments (Zuul) ● Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool) ● Smart commit gating (Zuul) ● config: everything as code ● Flexible workflow, based on reviewing and testing code prior to merging it ● Etherpad, pastebin, RepoXplorer and much more. ● Visualizing all the stuff through Kibana dashboard using opstools-ansible
- 18. Combining all to form RPM Factory ● rpmfactory : https://review.rdoproject.org ● The community platform for building RDO ● A Software Factory deployment branded for RDO ● Hosts all the distgit repositories + extra patches repositories ● All upstream changes acted upon with DLRN ● All distgit changes reviewed via Gerrit ● All distgit changes tested via Zuul + Jenkins + Nodepool + CBS ● New builds in CBS are automatically proposed when new releases are tagged upstream. ● Updating new versions of package for stable releases using bots ● Promoting new dependencies periodically in CBS through rdoinfo
- 20. Resulted in ● Master/ stable-branches/promoted yum repositories served https://trunk.rdoproject.org ● Consuming each bits by tripleo, kolla, packstack and puppet CI which run against different project’s submitted patches to find issues early ● Making it easy for installers to test against latest upstream code so that they can get released shortly after the services. ● Testing different stacks in different fashion through periodic and scenario jobs. ● Testing upstream gates jobs out of the gate using Weirdo. ● Leads to delivering more robust RDO cloud on time.
- 21. RDO stats for Ocata ● ~919 commits ● ~86 contributors ● RDO Ocata packages were available less than 12h after upstream release ● ~230 build failures caught by DLRN
- 22. RDO stats and learnings from Pike ● 1000+ commits ● 172 contributors ● Released in 2 days due to infra issues providing lessons to improve our infra ● Learnings: ○ Enable more automation ○ Better planning to cover PTO ○ Pipeline release speed matter
- 23. RDO Stats/Upcoming challenges for Queens ● 22 new contributors ● We are back on Track by releasing RDO same day. ● Auto Version updates for packages and dependencies ● Testing each spec changes on Fedora and Centos Builder ● Introduced Test cloud to test RDO Queens Milestone releases ● Upcoming challenges: ○ Moving to Zuul v3 ○ Python3 support
- 24. RDO Success Stories ● Cumulus Networks ● CERN ● Thoughtworks ● RDO Cloud on which develop OpenStack as well as running complete RDO infrastructure ● CentOS ● VEXXHOST
- 25. How to become a part of RDO community?
- 26. Contributing to RDO also leads to contributing to OpenStack.
- 27. How to contribute: ● Checkout https://www.rdoproject.org/easyfix/ ● Participate in #rdo meeting every wednesday at 08:00 P.M. (IST) on #rdo IRC channel ● Help on migrating stuff to zuulv3 and Python3 ● Improve RDO by participating in test days. ● Help us in improving RDO project documentation ● Look forward to improve Tripleo, Puppet-openstack, Kolla and OpenStack itself by taking a baby step ● Join #RDO{user, dev} mailing list
- 29. References: ● https://www.rdoproject.org/ ● https://softwarefactory-project.io/sf/welcome.html ● http://ara.readthedocs.io/en/latest/ ● https://review.rdoproject.org/sf/welcome.html ● http://cistatus.tripleo.org/ ● https://docs.openstack.org/infra/ ● https://www.openstack.org/videos/boston-2017/how-rdo-uses-software-factory-and-ansible ● https://www.rdoproject.org/user-stories/ ● https://blogs.rdoproject.org/ ● http://openstack.redhat.com/contribute/mailing-lists/ ● http://weirdo.readthedocs.io/en/latest/