![KuberDock CI today
- Automated Dev/QA environment. About 20 minutes build time.
TASK [debug] *******************************************************************
ok: [kd_master] => {
"msg": "http://XXX.XXX.XXX.XXX:5000 [user:XXX, password:XXX]"
}
TASK [debug] *******************************************************************
skipping: [kd_master]
PLAY RECAP *********************************************************************
kd_master : ok=69 changed=32 unreachable=0 failed=0
kd_node1 : ok=34 changed=9 unreachable=0 failed=0
- Based on Vagrant and Ansible
- Hosted in CL OpenNebula OR locally (VirtualBox)
- Engineer can multiple environments at time
- Customizable - skip_by_tag, hooks, dotfiles, etc.
- Developer friendly - code runs from source, multiple hacks are built-in.](https://image.slidesharecdn.com/whysocontinuous-160812133447/85/Why-so-continuous-13-320.jpg)
Why so continuous
- 1. Continuous Integration KuberDock, 10 Jun 2016
- 2. Agenda - Why so continuous? - Some huge examples - OpenStack - Spotify - World of Tanks - Ancestry - KuberDock CI today - Automated Dev/QA env - Integration Testing API & Tests - Jobs, Triggers - KuberDock CI in future - Parallelization -> scaling - CI as a Service - Quality Authority, sticking into the business processes - Continuous Delivery
- 3. Why so continuous? Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies to a shared mainline several times a day © Wikipedia Benefits: - Detect failure as earlier as possible - Repeatable, Stable & Incrementally improving - Catchall CI == Low mistake fear & Lower project entry barrier - Constant flow of changes to benefit users, no break & fix turnarounds - Fast regression == Fast release cycle Cons: - Expensive to Implement & Maintain
- 4. Yes, catchall CI is expensive, but Source: Bug fix cost over SW lifecycle
- 5. OpenStack CI OpenStack - Open source cloud platform ~ 20M Lines of code ~ 800 subprojects (different repos) ~ 2,5K Active contributors ~ 80K Gerrit reviews per cycle (6 months) OpenStack CI: Source: Zuul
- 6. OpenStack CI: the workflow
- 7. OpenStack CI, points of interest Their CI is fully declarative, lives under git - project: name: cinder github-org: openstack node: bare-precise tarball-site: tarballs.openstack.org doc-publisher-site: docs.openstack.org jobs: - python-jobs - python-grizzly-bitrot-jobs - python-havana-bitrot-jobs - openstack-publish-jobs - gate-{name}-pylint - translation-jobs - job: name: example-docs node: node-label triggers: - zuul builders: - git-prep - docs publishers: - scp: site: 'scp-server' files: - target: 'dir/ectory' source: 'build/html/foo' keep-hierarchy: true - console-log
- 8. OpenStack CI, points of interest Elastic Recheck With elastic-recheck now in place, contributors can: 1. Identify a pattern in the failure logs and visualize it in Kibana at http://logstash.openstack.org/ to search through a few weeks of logs to determine frequency. 2. Create a bug in our bug tracker for the error, add a comment to the bug with the exact query identified via Kibana, and a link to the logstash url for that query search. 3. Submit a simple YAML-based change to the elastic-recheck repository’s queries/ directory, which contains the list of bugs to track: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries. 4. Re-run tests only on the affected reviews / projects
- 9. OpenStack CI, points of interest Hardware vendors CI plugins TL;DR if you are hardware vendor writing your driver for an OpenStack component, you build your own CI server with that particular hardware, and link it to a common OpenStack gate. Gerrit has an event stream which can be subscribed to. Using this event stream, it is possible to test commits against testing systems beyond those supplied by OpenStack’s Jenkins setup. It is also possible for these systems to feed information back into Gerrit and they can also leave non-gating votes on Gerrit review requests. There are several examples of systems that read the Gerrit event stream and run their own tests on the commits on this page. For each patch set the third party system tests, the system adds a comment in Gerrit with a summary of the test result and links to the test artifacts.
- 10. Let’s talk about ~ 100M active users ~ 12K servers ~ Once a month desktop client releases - seamless ~ Clients on all mainstream mobiles and desktops & web. ~ All under CI/CD based on own container orchestration: Helios How Spotify does Continuous Delivery with Docker and Helios (video) Managing Machines at Spotify (tech blog)
- 11. Let’s talk about
- 13. KuberDock CI today - Automated Dev/QA environment. About 20 minutes build time. TASK [debug] ******************************************************************* ok: [kd_master] => { "msg": "http://XXX.XXX.XXX.XXX:5000 [user:XXX, password:XXX]" } TASK [debug] ******************************************************************* skipping: [kd_master] PLAY RECAP ********************************************************************* kd_master : ok=69 changed=32 unreachable=0 failed=0 kd_node1 : ok=34 changed=9 unreachable=0 failed=0 - Based on Vagrant and Ansible - Hosted in CL OpenNebula OR locally (VirtualBox) - Engineer can multiple environments at time - Customizable - skip_by_tag, hooks, dotfiles, etc. - Developer friendly - code runs from source, multiple hacks are built-in.
- 14. KuberDock CI today Integration tests are built on top of automated dev/qa env
- 15. KuberDock CI today Easy to use Integration Testing API
- 16. KuberDock CI today Integration tests triggers: - “integration” comment posted to review. - “dev-cluster” comment posted to review. - Night builds of both variants Unit tests triggers: - Run for each review - Run before merge (Workflow +1): 1-threaded rebase -> test -> merge
- 17. KuberDock CI in nearest future KuberDock Upgrades testing: - Allow to run the same test for clean cluster & upgraded cluster. - @clean_cluster() - @upgraded_cluster(from=’release’, to=’latest’) - Dedicated test to check if cluster workload survives after the upgrade - self.cluster.start(version=’release’) - pod = self.cluster.create_pod(healthcheck=True) - self.cluster.upgrade(version=’latest’) - pod.healthcheck()
- 18. KuberDock CI in nearest future Parallelization and test pipelines: - Tests are grouped in test pipelines - Pipelines are run in parallel to each other (they are differently-configured clusters) - Pipeline itself is divided to threads (one or more - different clusters, similar config) - Parallelized declaratively, constant arrangement, via pipeline name and thread number: - @pipeline(“clean”, thread=1) - @pipeline(“upgrade”, thread=1) - @pipeline(“non_floating_ips_clean”, thread=1) - @pipeline(“non_floating_ips_upgrade”, thread=1) - New tests are creating new pipelines or go to the new threads in existing ones - Integration test suite grows horizontally (more servers involved). - Full integration run time is kept around 40 minutes
- 19. KuberDock CI in future future - CI as a Service : everything is done through CI - Automated regression - Clusters for QAs for manual regression - Stable master cluster every hour - Release Candidate clusters - Release process tied to CI - Failure conditions (HA) testing (dropping nodes by one etc.) - Backup & restore testing (deploy -> check -> backup -> kill -> restore -> check) - Deployment configurations testing - Deployment performance testing - Cluster performance testing
- 20. Q & A
- 21. Links ● Understanding OpenStack CI ● OpenStack Gerrit-based workflow ● Elastic Recheck ● A CI/CD Alternative to Push and Pray for OpenStack ● How Spotify does Continuous Delivery with Docker and Helios ● Ancestry ● KuberDock automated DEV/QA env ● KuberDock integration tests