![Zabbix 7.0
What to do if "database upgrade failed"?
DON’T be hasty with restoring the backup DB copy!
Find the error message in the Zabbix server log file:
Find and eliminate root cause of the error (for example, recreate the missing index).
Run the Zabbix Server again to continue the upgrade process
6111:20190308:162806.987 starting automatic database upgrade
6111:20190308:162806.987 [Z3005] query failed: [1091] Can't DROP 'dservices_1'; check that
column/key exists [drop index dservices_1 on dservices]
6111:20190308:162806.987 database upgrade failed](https://image.slidesharecdn.com/upgrade-to-zabbix-70-240716140411-272efb6e/85/upgrade-to-zabbix-7-0-como-atualiza-lts1-45-320.jpg)
upgrade to zabbix-7 0 como atualiza lts1
- 1. Upgrade to Zabbix 7.0 LTS all our microphones are muted ask your questions in Q&A, not in the Chat use Chat for discussion, networking or applause Webinar
- 2. Zabbix 7.0 Zabbix 7.0 Focus on enterprise grade features enabling new use cases, better performance and out-of-the-box scalability Zabbix 7.0 focuses on: New widgets Zabbix Proxy in high-availability mode Synthetic end-user web monitoring Native two-factor authentication (MFA or 2FA) Asynchronous pollers Proxy memory buffer Centralized control of data collection timeouts Concurrent network discovery Custom input in frontend scripts Many other nice features…
- 4. Zabbix 7.0 Why upgrade Support of new functionality and bugfixes Improved performance and stability Improved security
- 5. Zabbix 7.0 New feature support Zabbix 6.2: Discovered host customization Zabbix 6.2: Active checks availability icon Zabbix 6.2: Separate groups for templates Zabbix 6.4: Cause and symptom problems Zabbix 6.4: Action log export to CSV Zabbix 6.4: Dashboard widgets Zabbix 7.0: Zabbix Proxy in high-availability mode Zabbix 7.0: Synthetic end-user web monitoring Zabbix 7.0: Proxy memory buffer Zabbix 7.0: New widgets Zabbix 7.0: Software update check Zabbix 7.0: Sending data to Zabbix server via Zabbix API
- 6. Zabbix 7.0 Improved stability and performance Zabbix 6.2: Faster configuration sync Zabbix 6.2: User macro cache Zabbix 6.2: Reload proxy configuration Zabbix 6.4: Optimized proxy configuration update Zabbix 6.4: Thread-based preprocessing workers Zabbix 6.4: Zabbix server support for older proxies Zabbix 7.0: Asynchronous pollers Zabbix 7.0: Concurrent network discovery Zabbix 7.0: Centralized control of data collection timeouts Zabbix 7.0: Faster reaction to host maintenance period update Zabbix 7.0: Automatic disabling of lost resources Zabbix 7.0: Faster permission checks
- 7. Zabbix 7.0 Improved security Zabbix 6.2: CyberArk Vault CV2 Zabbix 6.2: Multiple LDAP sources Zabbix 6.2: MD5 cryptography has been dropped completely for user passwords Zabbix 6.4: LDAP/SAML user provisioning Zabbix 6.4: Secure password change Zabbix 6.4: CSRF tokens Zabbix 7.0: Native two-factor authentication (MFA or 2FA) Zabbix 7.0: JIT user provisioning Zabbix 7.0: Logging of duplicate SNMPv3 Engine IDs
- 8. Life Cycle 2
- 9. Zabbix 7.0 Business service monitoring Release policy: • LTS (Long Term Support) release: 5.0, 6.0, 7.0 – for large environments, long term support, more thoroughly tested release. • Standard release: 6.2, 6.4 – Earlier access to new features. Major and minor releases: • 6.0, 6.2, 6.4, 7.0 - Major releases • 6.0.5, 6.2.18, 6.4.1, 7.0.1 - Minor releases
- 10. Zabbix 7.0 Life cycle Stable and unstable releases: • Even numbers: 6.2, 6.4, 7.0 – Stable releases • Odd numbers: 6.3, 6.5 - Unstable (alpha and beta) All the components contained within a single major release of Zabbix are 100% backwards compatible. Web frontend 7.0.0 + Server 7.0.1 + Proxy 7.0.0 + Agent 3.4.5 – OK (Minimum version of Agent 6.4 is recommended for active checks) Best practice is to have the same version on Frontend, Server and Proxy. No database changes within a single major release. Agents are backwards compatible: 1.4 – 7.0 agents compatible with 7.0.X server. Forward compatibility is not guaranteed: 7.2 – 7.4 agents might not be compatible with 7.0 server.
- 11. Upgrade types 3
- 12. Zabbix 7.0 Upgrade types Upgrade between minor versions: 3.4.7 → 3.4.15 4.0.0 → 4.0.5 4.0 → 7.0 4.4 → 7.0 Upgrade between major versions:
- 13. Zabbix 7.0 Which components to upgrade? The Zabbix minor upgrade procedure is very simple. You can upgrade a single or multiple components of Zabbix (for example, web frontend, proxy, Zabbix server). Zabbix agent upgrade is not mandatory.
- 14. Zabbix 7.0 Which components to upgrade? Zabbix major upgrade procedure is more complicated and requires prior preparation. Upgrade of all Zabbix components is required (Zabbix server, database, web frontend, proxy, Java Gateway, Web Service). Zabbix agent upgrade is not mandatory. (Minimum version of Agent 6.4 is recommended for active checks)
- 15. Zabbix 7.0 Minor upgrade procedure Read the release notes! Minor version upgrade: Make a backup copy (just in case) Upgrade the binary files and the web frontend files, restart the daemons No changes in the database?! (7.0.0 → 7.0.x) - not true from 6.0.11 when is backported some functionality from 6.2 $ dnf upgrade 'zabbix-*'
- 16. Zabbix 7.0 Minor upgrade procedure Fix the performance issues (if you have any). Otherwise the upgrade could take a considerable amount of time. Read the release notes!
- 17. Zabbix 7.0 Major upgrade procedure Major version upgrade: Make a backup copy Stop the Zabbix server Upgrade the binary files and the web frontend Start the Zabbix server Automatic database upgrade (2.2.0) Cumulative updates (2.2 → 7.0) Update rollback is not supported
- 18. Upgrade Notes 4
- 19. Zabbix 7.0 Upgrade notes 6.0 > 6.2 Check minimal DB version Check minimal PHP version DB triggers Primary keys User passwords in MD5 are not supported anymore Zabbix agent 6.2 – Active check icon CurlHttpRequest removed Templates
- 20. Zabbix 7.0 Upgrade notes 6.2 > 6.4 Check minimal DB version Check minimal PHP version It is recommended to use PHP 8.0 or newer, because PHP 7.4 is no longer supported by the vendor. Zabbix agent 6.4 – Active check The minimum required libssh version has been raised from 0.6.0 to 0.9.0 For SUSE Linux Enterprise Server (SLES) 15 the minimum supported service pack version is now SP4 DB triggers Automated database upgrade on proxies with SQLite Old numeric (float) value type is deprecated You can use outdated proxy in same LTS version for example 6.0 and 6.2 ConfigFrequency parameter for active proxies is now deprecated instead the ProxyConfigFrequency parameter must be used on both server (for passive proxies) and proxy (for active proxies). Note that if both ProxyConfigFrequency and ConfigFrequency are used, the proxy will log an error and terminate
- 21. Zabbix 7.0 Upgrade notes 6.4 > 7.0 Check minimal DB version Check minimal PHP version It is recommended to use latest version of PHP It is recommended to use latest version of DB Zabbix agent 6.4 – Active check DB triggers on MySQL Auditlog converted to hypertable on TimescaleDB When you are using MySQL partitioning, be careful with Auditlog table Old numeric (float) value type dropped Asynchronous pollers
- 22. Zabbix 7.0 Major upgrade: how NOT to perform an upgrade DO NOT read the release notes. DO NOT test the new version. DO NOT make a backup copy. $ dnf upgrade 'zabbix-*'
- 23. Zabbix 7.0 Why is "yum upgrade" bad? Zabbix User: We have an issue, the database size is growing really fast after upgrading from 1.8 to 3.0. Zabbix Team: The housekeeper has been disabled after the upgrade! Zabbix User: Java monitoring has stopped working after upgrading from 3.2 to 4.0. Zabbix Team: Java Gateway protocol has been changed, but Java Gateway has not been upgraded. Zabbix User: Zabbix restart takes too much time(Large amount of slow queries "select clock,ns,value from history"). Zabbix Team: MySQL parameter "index_condition_pushdown is enabled" (ZBX-10652). Zabbix User: Upgrade to the latest version took us 6 hours to perform (Downtime). Zabbix Team: There have been changes to large table structure.
- 24. Zabbix 7.0 Why do such situations occur? No knowledge about: changes, required upgrade steps. Larger Zabbix instance => more time to upgrade. Depends on: Zabbix version, database size, underlying hardware. Optional DB patches
- 25. Zabbix 7.0 How to avoid such situations? Read the release notes. Test new version before upgrade. Make a backup copy. Keep custom changes to minimum. Don’t make changes to database schema (partitioning).
- 26. Zabbix 7.0 How to plan the upgrade? Type of upgrade (minor – major). Changes in the structure of large tables (history, trends, events). Zabbix database size and performance. Complexity of the infrastructure (partitioning, High Availability cluster, replication). Customization (DB schema, custom PHP pages, patches, integrations, scripts, modules).
- 27. Zabbix 7.0 What causes errors? Changes in the DB schema: Removed or additional indexes (for example, with the goal to increase the performance). Partitioning misconfiguration. DB config misconfiguration.
- 28. Zabbix 7.0 What affects the upgrade duration? Zabbix DB size and performance. Changes in the structure of large tables (history*, trends*, events).
- 29. How to identify changes in large DB tables? Read the release notes! https://www.zabbix.com/documentation/3.2/en/manual/installation/upgrade_notes_320 history_text.id and history_log.id fields will be removed from the corresponding history tables during database upgrade. Depending on the history table size this process can be slow. https://www.zabbix.com/documentation/current/en/manual/installation/upgrade_notes_700 To successfully upgrade existing installations, you must rerun the postgresql/timescaledb/schema.sql script before starting Zabbix server. Zabbix server will log a warning, if started without running this script first. Zabbix 7.0
- 30. How to accelerate the upgrade in case of a large DB? Deletion of indexes for large tables can take hours (history_text/log). To accelerate Zabbix upgrade from hours to minutes - use temporary tables. Example for history_text: Create a temporary table (for example, history_text_tmp). Rename the existing table to history_text_old. Rename the new table history_text_tmp to history_text. Upgrade Zabbix (All the changes will be applied to empty table). Copy the data from the old table history_text_old to the new table history_text. Delete the old table history_text_old. Zabbix 7.0
- 31. Upgrade 5
- 32. Major upgrade: correct upgrade procedure Example Nº 1: Zabbix Upgrade: 5.0 → 7.0 (without proxy) DB size: 70GB (No customization or complexity) NVPS: 200 Acceptable downtime: Up to 2 hours Zabbix 7.0
- 33. Zabbix 7.0 Major upgrade: example №1 1. Read the release notes and take note of the important changes: 2. Make a backup copy of the configuration files, PHP files and scripts, etc. Example: $ cp /etc/zabbix/zabbix_server.conf /<backup directory>/ # Zabbix server config file $ cp /etc/zabbix/zabbix_agentd.conf /<backup directory>/ # Zabbix agent config file $ cp /usr/share/zabbix/alertscripts/* /<backup directory>/ # Alert scripts $ cp /usr/share/zabbix/externalscripts/* /<backup directory>/ # External scripts $ cp -R /usr/share/zabbix/ /<backup directory>/ # Web frontend PHP files $ cp /etc/httpd/conf/httpd.conf /<backup directory>/ # Apache config. files $ cp /etc/httpd/conf.d/zabbix.conf /<backup directory>/ # Zabbix/PHP parameters $ cp /etc/zabbix/web/zabbix.conf.php /<backup directory>/ # Zabbix frontend parameters.
- 34. Zabbix 7.0 Major upgrade: example №1 3. Make a backup copy of Zabbix DB 4. Stop Zabbix processes and update the packages $ screen -S backup $ mysqldump -uroot -p zabbix --extended-insert --single-transaction | bzip2 -9 > /<backup directory>/zabbix_db_backup-$(date "+%Y-%m-%d_%H.%M.%S").sql.bz2 $ systemctl stop zabbix-server $ rpm -Uvh https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-7.0-2.el9.noarch.rpm $ dnf upgrade zabbix*
- 35. Zabbix 7.0 Major upgrade: example №1 5. Start Zabbix server 7.0 process 6. Stop Zabbix processes and update the packages $ systemctl start zabbix-server $ tail -f /var/log/zabbix/zabbix_server.log 3152:20190308:175043.966 completed 100% of database upgrade 3152:20190308:175043.966 database upgrade fully completed
- 36. Zabbix 7.0 Major upgrade: correct upgrade procedure Example Nº 2: Zabbix Upgrade: 5.0 → 7.0 with 20 proxy servers DB size: 7 TB (partitioning) NVPS: 10000 Acceptable downtime: Up to 10 minutes Customized PHP pages, scripts and modules
- 37. Zabbix 7.0 Major upgrade: example №2 1. Read the release notes and take note of the important changes: 2. Create test environment to test compatibility of:
- 38. Zabbix 7.0 Major upgrade: example №2 3. Check the performance and optimize the DB settings (If necessary). Slow DB = Slow Upgrade! 4. Removal of internal events to accelerate the upgrade (changes in the events table 3.4 > 7.0) 5. Compare the Zabbix DB schema with the official one (for example, by using MySQL compare or a similar application). You need to understand if there are any customizations and if they can cause errors during the upgrade. 6. Create a backup copy of the configuration tables mysql> DELETE from events where source > 0 limit 10000; $ mysqldump –u<user> -p <database> --ignore-table=zabbix.history --ignore-table=zabbix.history_uint --ignore-table=zabbix.history_str --ignore-table=zabbix.history_log --ignore-table=zabbix.history_text --ignore-table=zabbix.trends --ignore-table=zabbix.trends_uint > zabbix_config_dump.sql
- 39. Zabbix 7.0 Major upgrade: example №2 7. Create a test database and import the initial schema/data into the Zabbix test environment. 8. Import the backup configuration table copy to the test environment 9. Perform the upgrade on test environment to obtain information regarding the upgrade duration and possible errors during the upgrade 10. Solve the issues detected during the test upgrade (if any were encountered) $ mysql -uzabbix -p zabbix_test < config_dump.sql $ systemctl stop zabbix-server $ rpm -Uvh https://repo.zabbix.com/zabbix/7.0/rocky/9/x86_64/zabbix-release-7.0-2.el9.noarch.rpm $ dnf upgrade zabbix-* $ systemctl start zabbix-server
- 40. Zabbix 7.0 Major upgrade: example №2 Continue with the production environment upgrade only if the test upgrade was performed successfully! 11. Make a backup copy of the configuration files, PHP files and scripts, etc. Example: $ cp /etc/zabbix/zabbix_server.conf /<backup directory>/ # Zabbix server config file $ cp /etc/zabbix/zabbix_agentd.conf /<backup directory>/ # Zabbix agent config file $ cp /usr/share/zabbix/alertscripts/* /<backup directory>/ # Alert scripts $ cp /usr/share/zabbix/externalscripts/* /<backup directory>/ # External scripts $ cp -R /usr/share/zabbix/ /<backup directory>/ # Web frontend PHP files $ cp /etc/httpd/conf/httpd.conf /<backup directory>/ # Apache config. files $ cp /etc/httpd/conf.d/zabbix.conf /<backup directory>/ # Zabbix/PHP parameters $ cp /etc/zabbix/web/zabbix.conf.php /<backup directory>/ # Zabbix frontend config file
- 41. Zabbix 7.0 Major upgrade: example №2 12. Make a backup copy of the Zabbix database 13. Temporarily disable all of the defined Actions, to avoid false notifications 14. Stop Zabbix processes and update the packages $ screen -S backup $ innobackupex -p<password> /<backup directory>/ $ innobackupex -p<password> --apply-log --use-memory=4G /<backup directory>/`date`/ $ systemctl stop zabbix-server $ rpm -Uvh https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-7.0-2.el9.noarch.rpm $ dnf upgrade zabbix-*
- 42. Zabbix 7.0 Major upgrade: example №2 15. Create temporary tables for history_text and history_log to avoid prolonged downtimes in the case of upgrading from version 3. 16. Start Zabbix server 7.0 process 17. Follow the database upgrade log entries $ systemctl start zabbix-server $ tail -f /var/log/zabbix/zabbix_server.log
- 43. Zabbix 7.0 Major upgrade: example №2 18. Check the Zabbix server performance after the upgrade (log file, unsupported items, queue, scripts) 19. Stop the proxy and delete the DB file in the case when SQLite DB is used (SQLite DB file upgrade is not supported) 20. Upgrade all of the proxy servers $ systemctl stop zabbix-proxy $ rm /tmp/zabbix_proxy.db $ rpm -Uvh https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-7.0-2.el9.noarch.rpm $ dnf upgrade zabbix-proxy* $ systemctl start zabbix-proxy
- 44. Zabbix 7.0 Major upgrade: example №2 21. Confirm that the proxies are running: - “Last seen” in Administration > Proxies - values and graphs from: “Template App Zabbix Proxy” 22. Enable Actions 23. Make sure latest «health» templates have been assigned to server/proxy (check the major version number at the end of link to fit with your version) 24. Upgrade Zabbix agents to obtain new functionality
- 45. Zabbix 7.0 What to do if "database upgrade failed"? DON’T be hasty with restoring the backup DB copy! Find the error message in the Zabbix server log file: Find and eliminate root cause of the error (for example, recreate the missing index). Run the Zabbix Server again to continue the upgrade process 6111:20190308:162806.987 starting automatic database upgrade 6111:20190308:162806.987 [Z3005] query failed: [1091] Can't DROP 'dservices_1'; check that column/key exists [drop index dservices_1 on dservices] 6111:20190308:162806.987 database upgrade failed
- 46. DEMO 6
- 47. php-fpm -v psql –V systemctl stop zabbix-server systemctl stop httpd dnf module switch-to php:8.2 –y systemctl stop postgresql-13 dnf install -y postgresql16-server timescaledb-2-postgresql-16 timescaledb-2-loader- postgresql-16 /usr/pgsql-16/bin/postgresql-16-setup initdb su – postgres cat /var/lib/pgsql/13/data/pg_hba.conf > /var/lib/pgsql/16/data/pg_hba.conf cat /var/lib/pgsql/13/data/postgresql.conf > /var/lib/pgsql/16/data/postgresql.conf /usr/pgsql-16/bin/pg_upgrade -b /usr/pgsql-13/bin -B /usr/pgsql-16/bin -d /var/lib/pgsql/13/data -D /var/lib/pgsql/16/data –k logout Zabbix 7.0 Demonstration
- 48. systemctl disable postgresql-13.service systemctl enable postgresql-16.service --now su - postgres psql –X c Zabbix ALTER EXTENSION timescaledb UPDATE; exit /usr/pgsql-16/bin/vacuumdb --all --analyze-in-stages reindexdb ./delete_old_cluster.sh rm -rf 13 delete_old_cluster.sh logout dnf remove postgresql13-* rm -rf /usr/pgsql-13/ Zabbix 7.0 Demonstration
- 49. rpm -Uvh https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-7.0-1.el9.noarch.rpm dnf clean all dnf update zabbix-* -y cat /usr/share/zabbix-sql-scripts/postgresql/timescaledb/schema.sql | sudo -u zabbix psql Zabbix systemctl start zabbix-server.service systemctl start httpd tail -f /var/log/zabbix/zabbix_server.log nano /etc/zabbix/zabbix_server.conf AllowUnsupportedDBVersions=1 systemctl restart zabbix-server.service tail -f /var/log/zabbix/zabbix_server.log timescaledb-tune --pg-config=/usr/pgsql-16/bin/pg_config --max-conns=100 systemctl restart postgresql-16.service Zabbix 7.0 Demonstration
- 50. Demonstration commands Latest how-to for upgrading to Zabbix 7.0 All necessary steps can be found in a separate guide on our wiki. https://www.initmax.com/wiki/zabbix-update-to-the-latest-version-7-0-lts/ EN https://www.initmax.cz/wiki/zabbix-aktualizace-na-posledni-verzi-7-0-lts/ CZ Support for the New Item Browser All necessary steps can be found in a separate guide on our wiki. https://www.initmax.com/wiki/installation-and-basic-usage-of-browser-item/ EN https://www.initmax.cz/wiki/instalace-a-zakladni-pouziti-pro-item-browser/ CZ Installation of Components and Configuration for Reporting All necessary steps can be found in a separate guide on our wiki. https://www.initmax.com/wiki/reporting-v-zabbixu-7-0/ EN https://www.initmax.cz/wiki/reporting-v-zabbixu-7-0/ CZ Zabbix 7.0
- 51. DEMO 7
- 52. Questions? 8
- 53. Zabbix 7.0 Contact us: Phone: +420 800 244 442 Web: https://www.initmax.cz Email: tomas.hermanek@initmax.cz LinkedIn: https://www.linkedin.com/company/initmax Twitter: https://twitter.com/initmax Tomáš Heřmánek: +420 732 447 184