How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions
0 likes419 views
This document discusses how MariaDB in Debian uses the Salsa-CI/Gitlab-CI infrastructure to ensure smooth upgrades and avoid regressions. It describes how the MariaDB packaging repositories were moved to Salsa.Debian.org and how a custom Gitlab-CI pipeline with 18 jobs over 5 stages was created to build, test, and simulate upgrades from various versions. This comprehensive continuous integration process helps catch issues early and improve the quality and reliability of MariaDB upgrades. Challenges in testing such a large and complex package are also discussed.
How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions
- 1. How MariaDB in Debian uses Salsa-CI/Gitlab-CI Ensure smooth upgrades and avoid regressions In Miniauditório at Tue 23rd July 2019 14:30-15:15 Follow @ottokekalainen to get link to slides!
- 2. Making use of new Debian infrastructure Salsa.Debian.org ● Debian’s own Gitlab instance ● First prototype in 2017, rolled-out big scale in 2018 ● See docs at https://wiki.debian.org/Salsa ● Run by Debian Salsa-Admin team ● MariaDB/MySQL packaging repositories moved from git.debian.org to salsa.debian.org in January 2018 Salsa-CI ● Debian packaging specific Gitlab-CI pipeline ● First version in summer 2018, very active development all the time ● See README at https://salsa.debian.org/salsa-ci-team/pipeline ● Run by Debian Salsa-CI team (I am a former member of it) ● Repositories for mariadb-10.1, mariadb-connector-c, galera-3 adopted it in August 2018
- 3. Salsa-CI is amazing If you are a Debian Developer, start using it now! Ensures that on every commit/push: ● Sources build and Debian packages are generated ● blhc: Build logs don’t have errors ● reprotest: Builds are reproducible (no randomness) ● autopkgtests: binaries can run ● lintian: static code analysis ● piuparts: packages install, uninstall and upgrade
- 4. ● Stop wasting your time on stupid mistakes and unnecessary prepare-upload-wait-check cycles. ● Stop wasting Debian archive resources with excess uploads. ● Give confidence to your contributors who can run CI on their merge requests. ● Give yourself a peace of mind, reduce stress and free up time to contribute more to Debian! ● Go from 10x engineer to 20x engineer!
- 5. Example: debian/gitlab-ci.yml contents in galera-3 Branch: master include: - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml Branch: buster include: - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml variables: RELEASE: 'buster'
- 6. Example: check Gitlab-CI config path and runner settings
- 7. Example: salsa.debian.org/mariadb-team/galera-3/pipelines Gitlab-CI offers real-time logs, graphs, artifacts download… Everything you need!
- 8. Example: Commit history and Merge Request statuses
- 10. Salsa-CI is very easy for a simple package See also Augustin Henze’s talk: debconf19.debconf.org/talks/148-salsa-ci-debian-pipeline-for-developers/
- 11. ..but MariaDB far from simple
- 12. 13 million lines of code...
- 13. One source package generates 25 binary packages... libmariadb-dev libmariadbclient-dev libmariadb-dev-compat libmariadb3 libmariadbd19 libmariadbd-dev mariadb-common mariadb-client-core-10.3 mariadb-client-10.3 mariadb-server-core-10.3 mariadb-server-10.3 mariadb-server mariadb-client mariadb-backup mariadb-plugin-connect mariadb-plugin-rocksdb mariadb-plugin-oqgraph mariadb-plugin-tokudb mariadb-plugin-mroonga mariadb-plugin-spider mariadb-plugin-gssapi-server mariadb-plugin-gssapi-client mariadb-plugin-cracklib-password-check mariadb-test mariadb-test-data
- 14. https://packages.debian.org/search?lang=en&suite=all&searc hon=names&keywords=mariadb-server https://packages.ubuntu.com/search?lang=en&suite=all&sear chon=names&keywords=mariadb-server Multiple upstream versions in maintenance in parallel in two distros each with multiple releases...
- 15. The configuration and data needs to persist and work over each upgrade... Default configuration /etc/mysql/ /etc/mysql/conf.d /etc/mysql/conf.d/mysql.cnf /etc/mysql/conf.d/mysqldump.cnf /etc/mysql/mariadb.conf.d /etc/mysql/mariadb.conf.d/50-client.cnf /etc/mysql/mariadb.conf.d/50-mysql-clients.cnf /etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf /etc/mysql/mariadb.conf.d/50-server.cnf /etc/mysql/my.cnf.fallback /etc/mysql/my.cnf /etc/mysql/mariadb.cnf /etc/mysql/debian-start /etc/mysql/debian.cnf Default data /var/lib/mysql/debian-10.1.flag /var/lib/mysql/mysql /var/lib/mysql/mysql/db.frm /var/lib/mysql/mysql/db.MYI /var/lib/mysql/mysql/db.MYD /var/lib/mysql/mysql/host.frm /var/lib/mysql/mysql/host.MYI /var/lib/mysql/mysql/host.MYD /var/lib/mysql/mysql/user.MYI /var/lib/mysql/mysql/user.MYD … /var/lib/mysql/mysql/user.frm /var/lib/mysql/aria_log_control /var/lib/mysql/aria_log.00000001 /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile0 /var/lib/mysql/performance_schema /var/lib/mysql/performance_schema/db.opt
- 16. Lot’s of direct users: 30 % of Debian popcon submitters
- 17. Hundreds of indirect users and dependencies that need to continue working... Packages that depend on MariaDB server or client directly apt-cache rdepends 'default-mysql-s*' 'mariadb-s*' | wc -l 96 apt-cache rdepends 'default-mysql-c*' 'mariadb-c*' | wc -l 73 Packages that depend on MariaDB client libraries apt-cache rdepends 'default-libmysql*' 'libmariadb*' | wc -l 194
- 18. MariaDB packaging custom Gitlab-CI pipeline
- 19. 18 jobs in 5 stages
- 20. Stage 1: build Build in target release plus one older (ensuring backporting is still possible).
- 21. Stage 2: test quality of built packages Quickly detect if the software malfunctions in an obvious way.
- 22. Stage 3: upgrade from current version in same release E.g. what will happen to users when the built version is uploaded to unstable (or stable if maintenance branch).
- 23. Stage 4: upgrade from the version in (old) stable releases E.g. what will happen when users run dist-upgrade from the previous stable or oldstable to the version being built now.
- 24. Stage 5: additional tests E.g. what will happen if users upgrade from MySQL 5.7 (which would not normally happen in a Debian dist-upgrade). Also lesser priority tests that are allowed to fail.
- 25. libmysql* to libmariadb* upgrade # Install all libmysql* available in Debian unstable - apt-get install -y pkg-config libmysqld-dev libmysqlclient-dev - pkg-config --list-all - pkg-config --cflags mysqlclient # mysqlclient.pc from original package - apt-get install -y ./libmariadb3_*.deb ./mariadb-common_*.deb - pkg-config --list-all - apt-get install -y ./libmariadb-dev_*.deb - pkg-config --list-all ... - apt-get install -y default-libmysqlclient-dev default-libmysqld-dev - ldconfig -p # | grep -e mariadb -e mysql - pkg-config --list-all - pkg-config --cflags --libs mysqlclient - pkg-config --cflags --libs libmariadb - pkg-config --cflags --libs mariadb
- 26. build mariadbclient consumer Python-MySQLdb - apt-get install -y pkg-config ./libmariadb-dev*.deb ./libmariadb3_*.deb ./mariadb-common*.deb - pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with - apt-get install -y python3-pip - pip3 install mysqlclient # Compiles module against libmysqlclient - apt-get purge -y libmariadb-dev # Not needed for run-time - python3 -c "import MySQLdb; print(MySQLdb.get_client_info())" Ensures we have a chance to notice if other software that used (legacy) libmysqlclient and are now automatically using libmariadb3 still works. Priceless: This caught upstream bugs what otherwise would have gone unnoticed for a long time since the “damage” is so subtle and invisible.
- 27. Explore it yourself! Gitlab, Salsa-CI and MariaDB evolves constantly salsa.debian.org/mariadb-team/mariadb-10.3
- 28. Challenges and peculiarities ● Custom gitlab-ci.yml has been developed now for a little over a year and grown to 550+ lines. ● Tries to cover all relevant install/upgrade scenarios. ● Docker images don’t have systemd running by default, must activate with ○ sed -i "s/101/0/g" -i /usr/sbin/policy-rc.d ● Build time max 2h ○ Slowness was tackled with ccache (and also upstreamed to Salsa-CI ● Build log max 4 MB ○ build command runs with ■ | tail -n 10000 ○ before build starts a a progress indicator is spawned ■ while true; do sleep 600; echo "10 minutes passed" >&2; done &
- 29. Next additions? ● Fix Jessie to latest upgrade test that regressied in early May 2019 with: ○ systemd-udevdstart-stop-daemon: unrecognized option '--notify-await' ○ Bug in Jessie or Jessie Docker image, not MariaDB ● Making RocksDB and TokuDB plugins build reproducibly ○ Can you help? ● Extend testing of MariaDB plugins and high-level features not covered by upstream MariaDB test suite ● Manipulate config files and database contents before starting upgrade to test how the maintainer scripts can recover from customized situations and heterogeneous environments ● Custom runner guarantee it is fast?
- 30. Thank you! @ottokekalainen Reminder of next session: Salsa and Salsa-CI BoF Room: Miniauditório Time: Jul 26 (Fri), 14:30