Transforming the Ceph Integration Tests with OpenStack
0 likes205 views
This document discusses transforming Ceph tests to use OpenStack. It describes running unit tests locally but running integration tests on OpenStack instances. Developers can now run integration tests on their own OpenStack tenant without waiting for resources. Specifying resources for the OpenStack machines makes the tests more self-service. Future improvements include better multi-cloud support and making archival and setup more convenient.
Transforming the Ceph Integration Tests with OpenStack
- 1. Transforming the Ceph tests with OpenStack Loic Dachary Artisan Logiciel Libre 2015/10/27
- 2. Ceph developers population Elite: Smart people : Sage, Sam, Josh... Large companies : Red Hat, SUSE, Intel... Normal: You and me Non-profit : Tetaneutral, FSF...
- 3. Normal developer loop 2014 Setup Ceph $1 Hit a bug and report it $10 Fix the bug $1000 Rebuild the packages $10 Run tests N/A Upgrade the package N/A
- 4. Normal developer loop 2015 Setup Ceph $1 Hit a bug and report it $10 Fix the bug $1000 Rebuild the packages $10 Run tests $100 Upgrade the package $1
- 5. Ceph development lifecycle commit => test => review => merge => test => release
- 6. Tests commit => test => review => merge => test => release test == unit tests => packages => integration tests
- 7. Unit tests Unit tests and functional tests : not too long, single machine, no network, not root Make check local Make check bot Stable and under surveillance
- 8. Debugging unit tests make check bot fails => run locally => fix => push --force
- 9. Integration tests Requires more than one machine or Takes more than a few minutes Each job taks ~2 machines and takes ~2 hours Before a release ~2,000 jobs are run and failures analyzed
- 10. Running integration tests 2014 Push a branch on the official repository Wait for gitbuilder to create a package teuthology-suite to add the test in the sepia lab
- 11. Integration tests 2015 teuthology-openstack with an OpenStack tenant
- 12. Debugging integration tests 2015 teuthology fails => debug from log or pause the test => fix => push --force
- 13. Commit review 2014 Developer: Pull request Automatic: Unit test pass Reviewer: Review the commit Reviewer: Add needs-qa label Reviewer: Merge and push in a testing branch Reviewer: Wait for gitbuilders Reviewer: Run teuthology-suite Reviewer: Sort the teuthology-suite results
- 14. Commit review 2015 Developer: Pull request Automatic: Unit test pass Reviewer: Review the commit Developer: Run teuthology-openstack Developer: Sort the teuthology-suite results Developer: Fix and re-push
- 15. OpenStack prerequisites Flavors with 16 cpus and 32GB RAM Floating IPs (optional) Volumes (optional) Security groups (optional) Tested with Havana, Juno & Kilo clusters
- 16. Adding OpenStack workers add openstack: to ~/.teuthology.yaml source openrc.sh teuthology-worker –tube openstack teuthology-suite –machine-type openstack
- 17. self-service teuthology- openstack use a tenant with no pets source openrc.sh create a passwordless keypair ~/myself.pem teuthology-openstack –suite rados watch http://120.43.25.39:8081/ teuthology-openstack --teardown
- 18. --key-name –key-filename openstack keypair create myself teuthology-openstack --key-name myself --key-filename ~/myself.pem
- 19. --throttle --simultaneous-jobs teuthology-openstack --throttle 30 --simultaneous-jobs 200
- 21. --archive-upload --upload teuthology-openstack –archive-upload ubuntu@teuthology- logs.public.ceph.com:./ --upload
- 22. Resources description openstack: - machine: disk: 40 # GB ram: 8000 # MB openstack: - machine: ram: 32000 # MB
- 23. Resources description openstack: - machine: disk: 40 # GB ram: 32000 # MB
- 24. Resources description defaults openstack: machine: disk: 20 # GB ram: 8000 # MB cpus: 1 volumes: count: 0 size: 1 # GB
- 25. Caveats mulitple OpenStack provider support --verbose to see feedback archival with –upload would be more convenient as a standalone action setting up teuthology should be done when compiling Ceph locally
- 26. Commit review 20XX Developer: Pull request Automatic: Unit test and integration tests pass, display coverage of the commit modifications and cross reference the results in the corresponding issue Reviewer: Review the commit