![Nodepool: конфигурация
/etc/nodepool/secure.conf
1) Создать пользователя jenkins, настроить ssh авторизацию и
sudo
2) apt-get update и установить dev пакеты
3) git clone git://libvirt.org/libvirt.git в /opt/git
/etc/nodepool-scripts/my-setup.sh
/etc/nodepool/nodepool.yaml
[database]
dburi=mysql+pymysql://nodepool:passwd@localhost/nodepooldb
[jenkins "jenkins1"]
user=jenkins
apikey=f828c94dbd53b7e2a055112ebbc90ea8
credentials=28755fdb-244a-4fcf-baee-496031323c12
url=http://10.94.136.33:8080/](https://image.slidesharecdn.com/qaconf-160426114215/85/Openstack-Third-Party-CI-and-the-review-of-a-few-Openstack-Infrastructure-projects-10-320.jpg)
![Диагностика gearman: status
request
Status line fmt: Function <enqeued> <executing> <Nworkers>
root@test-nodepool:/etc/jenkins_jobs/config# telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
build:smoke-libvirt-v1.3.1-maint 0 0 1
set_description:10.94.136.33 0 0 1
build:smoke-libvirt-master 0 0 1
stop:10.94.136.33 0 0 1
build:smoke-libvirt-v1.3.1-maint:dib-trusty-label 0 0 1
build:smoke-libvirt-master:dib-trusty-label 0 0 1
.
^]
telnet> Connection closed.](https://image.slidesharecdn.com/qaconf-160426114215/85/Openstack-Third-Party-CI-and-the-review-of-a-few-Openstack-Infrastructure-projects-31-320.jpg)
Openstack Third-Party CI and the review of a few Openstack Infrastructure projects
- 1. Openstack Third-Party CI и обзор компонент тестовой инфраструктуры Евгений Антышев, Virtuozzo eantyshev@virtuozzo.com skype: Evgeny.Antyshev
- 2. Более 600 активных разработчиков(!) 54 команды, у каждой в среднем 10 проектов, от 1 до 138 (Infrastructure) http://git.openstack.org/cgit/openstack/governance/plain/reference/projects.yaml 1. Основной язык -- Python. Затем: Shell, Puppet. Для данных: Yaml, ini-config, json. 2. Общая структура репозиториев: README.md, testr unittests, tox virtualenv wrapper 3. Общий процесс разработки: git, gerrit review
- 4. Задачи third-party CI: 1)Верификация входящих патчей на своем окружении 2)Верификация third-party компонент как необходимое условие для их принятия сообществом 3)Стабильность third-party CI создает хорошую репутацию в сообществе
- 5. Openstack CI scaling “Jenkins”: upstream CI robot 780 проектов ~900 workers ~20k events/day Virtuozzo CI, Virtuozzo Storage CI 3 проекта 3-6 workers 150 events/day
- 6. “Легенда”: удаленное тестирование кода libvirt • Разворачивание и обновление тестовых сред с Ubuntu 14.04 LTS (nodepool, diskimage-builder) • Конфигурация тестовых задач для Jenkins (Jenkins Jobs Builder) • Управление запуском тестов (gearman)
- 7. Nodepool Управление пулом тестовых серверов и загрузочными образами в Openstack облаке
- 8. nodepool Jenkins + plugins: ZMQ, Node and Label, Gearman WorkerN “Trusty-label” qemu-kvm ssh REST API “Trusty-label” image Ubuntu 14.04 (Trusty Tahr) base image my-setup.sh 1) Create/update images 2) Deploy nodes for Jenkins and utilize when it finishes with them Gearman server Provider cloud
- 9. Nodepool: установка 1. PyPi # pip install nodepool 2. Git sources # git clone git://git.openstack.org/openstack-infra/nodepool.git ../nodepool # pip install -r requirements.txt ../nodepool # pip install . 3. puppet-nodepool project # git clone git://git.openstack.org/openstack-infra/puppet-nodepool.git Use README
- 10. Nodepool: конфигурация /etc/nodepool/secure.conf 1) Создать пользователя jenkins, настроить ssh авторизацию и sudo 2) apt-get update и установить dev пакеты 3) git clone git://libvirt.org/libvirt.git в /opt/git /etc/nodepool-scripts/my-setup.sh /etc/nodepool/nodepool.yaml [database] dburi=mysql+pymysql://nodepool:passwd@localhost/nodepooldb [jenkins "jenkins1"] user=jenkins apikey=f828c94dbd53b7e2a055112ebbc90ea8 credentials=28755fdb-244a-4fcf-baee-496031323c12 url=http://10.94.136.33:8080/
- 11. Nodepool.yaml: header script-dir: /etc/nodepool-scripts elements-dir: /etc/nodepool-elements images-dir: /opt/nodepool_dib cron: check: '*/15 * * * *' cleanup: '*/1 * * * *' image-update: '14 2 * * *' zmq-publishers: - tcp://localhost:8888
- 12. Nodepool.yaml Jenkins label → image name labels: - name: trusty-label image: trusty1 min-ready: 1 providers: - name: openstack001-test
- 13. nodepool.yaml Cloud providers providers: - name: openstack001-test username: 'admin' password: '5fcf6292e9a240e0' auth-url: 'http://10.94.2.99:5000/v2.0/' project-name: 'admin' max-servers: 2 images: - name: trusty1 base-image: trusty setup: my-setup.sh min-ram: 2048 username: jenkins private-key: /opt/nodepool/jenkins_key
- 14. Nodepool command line > nodepool hold 123 > nodepool delete 123 > nodepool image-update all trusty1
- 15. Nodepool: создание загрузочных образов Snapshot images Base image Temporary VM my-setup.sh Take a snapshot diskimage-builder (DIB) images Base image qemu-nbd mount Prep. in chroot Convert Upload to provider
- 16. Преимущества diskimage- builder: 1) Можно запускать в VM 1) Более структурированный способ задания конфигурации 1) Легкое добавление готовых элементов в конфигурацию
- 17. diskimage-builder: типичные задачи для автоматизации ■адреса репозиториев ■пользователи, ключи ssh ■установка и конфигурация приложений ■разделы фс, настройки загрузки (growroot и пр.) ■кеширование git репозиториев и пр.
- 18. DIB stages & elements Stages: root.d, extra-data.d, install.d (in chrooted env.), finalise.d, etc. Elements: • ubuntu-minimal: base OS • vm: preparation to boot in QEMU • simple-init: boot scripts to enable DHCP for NICs • growroot: expand root FS to cover available disk space
- 19. DIB element “my-slave” (derived from my-setup.sh) /etc/nodepool-elements/my-slave/ package-installs.yaml -- sshd, dev packages extra-data.d/ 50-jenkins-pubkey -- copy key from host to mounted image install.d/ 50-jenkins-user 60-cache-libvirt finalise.d/ 99-nodepool-dir -- nodepool needs /etc/nodepool
- 20. /etc/nodepool/nodepool.yaml: diskimages: - name: ubuntu-trusty elements: - ubuntu-minimal - vm - simple-init - growroot - my-slave release: trusty env-vars: TMPDIR: /opt/dib_tmp DIB_IMAGE_CACHE: /opt/dib_cache /var/log/nodepool.log: disk-image-create -x -t qcow2 --no-tmpfs --qemu-img- options 'compat=0.10' -o /opt/nodepool_dib/42 ubuntu-minimal vm simple- init growroot my-slave … Image file /opt/nodepool_dib/42.qcow2 created… nodepool.yaml: diskimages
- 21. Jenkins Job Builder Конфигурация как код Yaml вместо Xml Шаблоны, группы, макросы
- 22. JJB builder: copy-update-source - builder: name: copy-update-source builders: - shell: | mv /opt/git/libvirt $WORKSPACE cd libvirt git clean -f -x git fetch -f --recurse-submodules=yes origin {branch} git checkout FETCH_HEAD
- 23. JJB builder: test-libvirt - builder: name: test-libvirt builders: - shell: | cd libvirt ./autogen.sh --system make sudo make install libvirtd --version sudo libvirtd -d virsh list --all
- 24. JJB template - job-template: name: smoke-libvirt-{branch} node: trusty-label || dib-trusty-label builders: - copy-update-source: branch: '{branch}' - test-libvirt publishers: - email: recepients: 'eantyshev@virtuozzo.com'
- 25. JJB project - project: name: libvirt branch: - master - v1.3.1-maint jobs: - smoke-libvirt-{branch}
- 26. Jenkins view
- 27. Gearman Сервис распределения задач и client/worker API http://gearman.org “We on Openstack infrastructure team use Jenkins extensively. Our jenkins servers, at peak load, runs 20,000+ jobs per day. At that load we require many jenkins slaves (900+) to process all of those build jobs. We have found that our requirement was pushing Jenkins beyond it's limits therefore we've decided to create the Gearman Plugin to support multiple Jenkins masters. The gearman plugin was designed to support extra slaves, allow load balancing of build jobs, and provide redundancy.” https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
- 28. Gearman workflow example 1. Worker registers “reverse” function on Gearman server. 2. Client sends “reverse” request and parameters. 3. Gearman server chooses the worker and assigns the job. 4. Worker sends result to Client via Server.
- 29. Ubuntu 14.04: apt-get install gearman-job-server /etc/nodepool/nodepool.yaml: gearman-servers: - host: localhost Jenkins -> Configure System -> Enable Gearman Установка gearman-server, Gearman plugin (client)
- 30. Gearman interactions in “libvirt CI” gearmand Gearman Client (Python API) Jenkins Gearman plugin (Java API) Nodepool (status TCP requests + images tasks) Gear worker
- 31. Диагностика gearman: status request Status line fmt: Function <enqeued> <executing> <Nworkers> root@test-nodepool:/etc/jenkins_jobs/config# telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status build:smoke-libvirt-v1.3.1-maint 0 0 1 set_description:10.94.136.33 0 0 1 build:smoke-libvirt-master 0 0 1 stop:10.94.136.33 0 0 1 build:smoke-libvirt-v1.3.1-maint:dib-trusty-label 0 0 1 build:smoke-libvirt-master:dib-trusty-label 0 0 1 . ^] telnet> Connection closed.
- 32. Job requests via Gear client Python implementation: http://pypi.python.org/pypi/gear https://github.com/zaro0508/gearman-plugin-client >python gear_client.py -s localhost -p 4730 --function build:smoke-libvirt-master:trusty-label --params '{"OFFLINE_NODE_WHEN_COMPLETE":"true"}'
- 33. Zuul Планирование тестовых запусков для per-patch CI, project gating для gerrit http://docs.openstack.org/infra/zuul/
- 34. Zuul pipelines Independent check, post- merge, periodic Dependent gate HEAD 1 HEAD 2 HEAD 3 HEAD 1 2 3
- 35. Independent pipelines Dependent pipeline http://status.openstack.org/zuul/
- 36. Материалы “libvirt CI”: https://github.com/eantyshev/libvirt_ci Полезные ссылки: http://docs.openstack.org/infra/manual/ http://docs.openstack.org/infra/nodepool/ https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin http://docs.openstack.org/infra/jenkins-job-builder/ Спасибо за внимание! Вопросы?
- 38. Система ревью для git Робот, управляющий запуском тестов (gear python client) Сервис очередей для распределения задач Jenkins master (Gearman worker) Jenkins slave (рабочие среды) Создание и удаление рабочих сред, обновление образов (Gearman queue monitoring)
- 39. /etc/jenkins-jobs/config/libvirt.yaml: - job-template: name: smoke-{name}-{branch} node: trusty-label || dib-trusty-label JJB: node labels
- 40. smoke-libvirt job, take 2: split builders - job: name: smoke-libvirt project-type: freestyle node: trusty-label builders: - shell: | mv /opt/git/libvirt $WORKSPACE cd libvirt git clean -f -x git fetch -f --recurse-submodules=yes origin master git checkout FETCH_HEAD - shell: | ./autogen.sh --system make sudo make install libvirtd --version sudo libvirtd -d virsh list --all publishers: - email: recepients: 'eantyshev@virtuozzo.com'
- 41. ● root.d: Download cloud image (qcow2) mount by qemu-nbd, submounts: /dev, /dev/pts, /sys, /proc + hooks dir ● environment.d: Set default environment vars ● extra-files.d: copy files to mounted image, f.e. ssh keys Go chroot under mounted path ● pre-install.d, install.d, post-install.d: Executed in chroot environment: set repo mirrors, install packages, configure apps Copy root content to temp. Unmount amended image. Make clean raw image file ● block-device.d: (auxiliary) Partition loopback block device Create filesystem, mount the image (+/proc,dev,sys submounts), copy rootfs contents from temp ● finalise.d: install/configure bootloader, other tools ● cleanup.d: cleanup temporary repos, unblock daemons Unmount, convert from raw to qcow2 DIB stages
- 42. smoke-libvirt job, take 1 Недостатки: copy-paste when >1 job Неструктурированная логика в builders 1) Макрос copy-update-source с параметром branch и макрос test-libvirt 2) Job -> Job Template + Project 3) “project-type: freestyle” -> defaults.yaml - job: name: smoke-libvirt project-type: freestyle node: trusty-label builders: - shell: | mv /opt/git/libvirt $WORKSPACE cd libvirt git clean -f -x git fetch -f --recurse-submodules=yes origin master git checkout FETCH_HEAD ./autogen.sh --system make sudo make install libvirtd --version sudo libvirtd -d virsh list --all publishers: - email: recepients: 'eantyshev@virtuozzo.com'
- 43. JJB defaults - defaults: name: global project-type: freestyle concurrent: true wrappers: - timeout: timeout: 30 fail: true - timestamps logrotate: daysToKeep: 7 numToKeep: -1 artifactDaysToKeep: -1 artifactNumToKeep: -1
- 44. type: patchset-created, patchSet.ref: 'refs/changes/38/307638/1' type: comment-added, comment: “Looks good overall, couple of ....” type: patchset-created, patchSet.ref: ‘refs/changes/78/216378/38’ type: comment-added, comment: “blablabla...” zuul-server (scheduler) zuul-merger (git server) > gerrit event-stream Gearman server merger:merge 0 0 1 merger:update 0 0 1 Job requests: merger:merge args{gerrit_url, project, ref}* merger:update args{gerrit_url, project} Merge job complete: Zuul ref: commit id git clone <url> /var/lib/zuul/git/<project> git merge <ref> In Jenkins job: git fetch $ZUUL_URL $ref Zuul workflow
- 45. Что дальше..? • Как запускать тесты автоматически? • Как регулировать число тестовых нод в зависимости от нагрузки? • Как масштабировать CI на несколько Jenkins серверов?
- 46. - job: name: test-example builders: - shell: ‘echo Success!’ <?xml version="1.0" encoding="utf-8"?> <project> <actions/> <description><!-- Managed by Jenkins Job Builder --></description> <keepDependencies>false</keepDependencies> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <concurrentBuild>false</concurrentBuild> <canRoam>true</canRoam> <properties/> <scm class="hudson.scm.NullSCM"/> <builders> <hudson.tasks.Shell> <command>echo Success!</command> </hudson.tasks.Shell> </builders> <buildWrappers/> </project> Yaml to Xml > jenkins-jobs test sample-job.yaml