OpenNebulaConf 2014 - CentOS, QA and OpenNebula - Christoph Galuschka
0 likes569 views
- Christoph Galuschka uses OpenNebula to automate QA testing of CentOS releases through continuous integration (CI). The CI environment consists of 3 servers running OpenNebula and Jenkins to instantiate virtual machines (VMs) and run automated tests on different CentOS versions. - The t_functional Git repository contains over 430 package and role tests that are run daily by Jenkins on current CentOS VMs. Community contributions of new tests are merged after review. Test results are available on ci.dev.centos.org. - OpenNebula is also used to control VMs hosting CentOS websites and services across two additional data centers, demonstrating its use beyond just CI/QA automation.
![KS-Sample for prjrd.net
sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init
sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth*
rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules
for f in /boot/grub/grub.conf; do
/bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f}
done
rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd
cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF
/sbin/restorecon -Rv /root/.ssh
if [ ! -e /var/update_done ]
then
yum update -y
touch /var/update_done
chkconfig sshd on
chkconfig auditd on
reboot
fi
EOF
%end
● This kickstart creates an image with disabled ssh-server. After boot it pulls
all updates, enables ssh and reboots once (ensuring current kernel for all
things to come for this VM).](https://image.slidesharecdn.com/centosqaandopennebula-christophgaluschka-141210051540-conversion-gate01/85/OpenNebulaConf-2014-CentOS-QA-and-OpenNebula-Christoph-Galuschka-12-320.jpg)
OpenNebulaConf 2014 - CentOS, QA and OpenNebula - Christoph Galuschka
- 1. CentOS, QA and OpenNebula Opennebula Conference 2014 Berlin December 2nd to 4th Christoph Galuschka tigalch@tigalch.org
- 2. Who am I ● Christoph Galuschka, age 37 ● from Innsbruck, Austria ● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT operations department ● Using Linux since 1998, CentOS since 2005 ● Part of the CentOS-QA-Team since April 2012 ● Mainly working on release-QA, t_functional git repo for automated testing and the testing infrastructure, wiki and forums
- 3. CentOS CI ● created in june 2011 ● an approach to automate QA-release testing ● create a set of tests to run for each point release QA-cycle ● use git as a common repository for other users (yes you too) to contribute new tests ● currently contains more then 430 scripts/functional/trademark tests for more than 110 packages ● additionally uses the Linux Test Project ● upcoming: to test every update before public release
- 4. History ● First setup of CI: lost from memory :) and in a different DC ● Second setup - move to opennebula ● started with 4.0, 4.2 later (RPMs from CentOS-Developers) ● 4.8 since November 2014 ● only “issue” during update: ruby gem ● ‘/usr/share/one/install_gems’ vs. rpm
- 5. Setup I ● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each ● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors are solely used for test VMs
- 6. Setup II
- 7. Setup III ● OpenNebula is used as the cloud controller ● The sunstone interface is available on Hypervisor1 ● All VMs instantiated only available on the private network ● Datastore transport → SSH ● Hosts → KVM ● The ci.dev.centos.org site runs as VM in that setup ● nginx is used to forward http(s) from the public internet to the VM ● local mirror for all CentOS-releases ● mimics the setup of mirror.centos.org for all VMs ● Jenkins is used to handle all running tests/VMs
- 8. Jenkins ● 4 different testgroups ● docker image (C6/C7 only), t_functional, LTP and reimzul ● run on a daily basis ● Infrastructure is hosted by GoDaddy
- 9. Jenkins II ● All tests run the current versions of CentOS (C5.11, C6.6, C7.0) Docker image ● pulls and runs current C5/6/7 docker image ● checks installation and command execution inside image LTP ● Linux Test Project1 ● comprehensive checks of basic linux functionality t_functional ● functional tests of packages ● also checks ™ issues if valid ● 14 contributors ● C7 running since Nov. 7th reimzul ● called on de-mand via IRC ● used for verifiying merge requests 1) visit http://ltp.sourceforge.net/
- 10. Images used for CI ● Created by Karanbir Singh and Jaime Melis ● Creates images for C5 and C6 with generic kickstarts and configurations ● Images are custombuilt on the fly and are downloadable afterwards ● C7 image comes from cloud. centos.org
- 11. KS-Sample for prjrd.net install url --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ lang en_US.UTF-8 keyboard uk network --device eth0 --bootproto dhcp rootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI. firewall --service=ssh authconfig --enableshadow --passalgo=sha512 --enablefingerprint selinux --enforcing timezone --utc Europe/London bootloader --location=mbr --driveorder=sda repo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100 repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100 zerombr yes clearpart --all --initlabel part /boot --fstype ext3 --size=250 part pv.2 --size=5000 --grow volgroup VolGroup00 --pesize=32768 pv.2 logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512 reboot %packages @Base @Core opennebula-context %end %post --log=/root/post.log --nochroot ...
- 12. KS-Sample for prjrd.net sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth* rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules for f in /boot/grub/grub.conf; do /bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f} /bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f} /bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f} done rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF /sbin/restorecon -Rv /root/.ssh if [ ! -e /var/update_done ] then yum update -y touch /var/update_done chkconfig sshd on chkconfig auditd on reboot fi EOF %end ● This kickstart creates an image with disabled ssh-server. After boot it pulls all updates, enables ssh and reboots once (ensuring current kernel for all things to come for this VM).
- 13. introducing t_functional ● Git repo hosting all the functional tests (packages, roles) ● Using #!/bin/bash (but other scripting languages are possible!) ● Hosted on gitorious and git.centos.org (for QA) ● Community driven (yes, you too) ● Daily results at: http://ci.dev.centos.org ● https://git.centos.org/summary/sig-core!t_functional.git ● https://gitorious.org/testautomation/t_functional
- 14. Distro and functional tests ● Is release X.y installable? ● Through ISO? ● Through network (nfs/http/ftp) + pxe? ● Mixed (netinstall.iso + network)? ● Usb image? ● UEFI vs Bios? ● Are all packages in the distro doing what they're supposed to do ? ● Still working with different sets of packages (php vs php53 for example in CentOS 5.x)
- 15. t_functional git workflow gitorious Clone & change Request to merge Jenkins CI merge request Fix it No? No? gitorious push git.centos.org pull QA-Env
- 16. further opennebula usage ● Two further setups in DCs in London ● controls four hosts in both cases ● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy, some internal stuff ● www and forum site to come ● uses 4.8 ● GlusterFS over Infiniband ● more on that: http://arrfab.net/blog/?p=485
- 17. Any Questions? Thank you for your time