33616611930205162156 upgrade internals_19c

And so much more ...
Mike Dietrich, Daniel Overby Hansen, Frederick Alvarez
Database Upgrade and Cloud Migrations
Oracle Corporation
Copyright © 2021, Oracle and/or its affiliates
1
Upgrade Internals
Photo
by
Rod
Long
on
Unsplash

Distinguished Product Manager
Database Upgrade
and Migrations
Mike Dietrich
https://MikeDietrichDE.com
@MikeDietrichDE
mikedietrich

Senior Principal Product Manager
Cloud Migration
Daniel Overby Hansen
https://dohdatabase.com
@dohdatabase
dohdatabase

Principal Software Developer
Database Upgrade Team
Frederick Alvarez
https://donhk.dev
@idonhk
frealvar

https://MikeDietrichDE.com https://DOHdatabase.com https://donhk.dev/
Blogs
5

6
Photo
by
Daniel
Cheung
on
Unsplash
Database
Upgrade
Basics

Database Upgrade
Why upgrade?
Photo
by
SpaceX
on
Unsplash

Lifetime Support Policy
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
Oracle 18
(12.2.0.2)
EXTENDED
Oracle 12.1
EXTENDED
Oracle 19
(12.2.0.3)
EXTENDED
Waived EXTENDED
Oracle 11.2 MARKET
DRIVEN
Oracle 12.2.0.1 LIMITED
Premier Support Waived Extended Support Paid Extended Support Market Driven Support Limited Error Correction
Oracle 21

Innovation
EXTENDED
Long Term Support
Innovation
Premier Support Paid Extended Support
Release Types | Long Term Support vs Innovation Releases
Long Term Support Release
• 5 years of Premier Support followed by 3 years of Extended Support
Innovation Release
• 2 years of Premier Support, but there is no Extended Support
• MOS Note: 742060.1 - Release Schedule of Current Database Releases

Oracle 12.2 Release Family
Includes:
• Oracle 12.2.0.1, Oracle 18c (12.2.0.2), Oracle 19c (12.2.0.3)
• MOS Note:742060.1 - The Single Source of Truth
• MOS Note:161818.1 - Releases Support Status Summary
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
Oracle 18
(12.2.0.2)
EXTENDED
Oracle 12.2.0.1
Oracle 19
(12.2.0.3)
LIMITED
Oracle 21

Database Upgrade
Why is Upgrade so different?
Photo
by
Nick
Fewings
on
Unsplash

Why Upgrade is Different | Short or Long?
Things that matter a lot
• Number of installed components
• Size & complexity of dictionary
• Some feature/version combinations
Things that matter a little
• CPU and disk speed
• SGA/PGA
Things that don't matter
• Amount of user data
select COMP_ID, STATUS from CDB_REGISTRY;
select count(*) from OBJ$;
Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle Workspace Manager 00:01:01
Oracle Enterprise Manager 00:10:13
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Oracle Multimedia 00:07:43
Gathering Statistics 00:04:53
Total Upgrade Time: 00:52:01
Component HH:MM:SS
Oracle Server 00:16:17
JServer JAVA Virtual Machine 00:05:19
Oracle XDK 00:00:48
Oracle Text 00:00:58
Oracle XML Database 00:04:09
Oracle Database Java Packages 00:00:33
Gathering Statistics 00:02:43
Total Upgrade Time: 00:30:47

Why Upgrade is Different | Component Invalid or Removal?
Remove and Clean Up Components
• If you plan to do this, do it BEFORE the upgrade
• Especially components which don't exist in the new release should be removed beforehand

Why Upgrade is Different | DDL Workload
Heavy DDL workload
• A trivial upgrade from 11.2.0.4 to 19c includes approximately
• 2,500 ALTERs (mostly tables, also types and users)
• 16,500 CREATE OR REPLACEs (views, types, synonyms, procedures, packages, functions)
• 10,200 GRANT statements
• 1150 CREATE TABLE statements
• 360 CREATE INDEX statements
• And more…
Almost exclusively in SYSTEM,
partially also in SYSAUX tablespaces

Why Upgrade is Different | Upgrade Mode
STARTUP UPGRADE limits many aspects of the database
• Requires SYSDBA privilege
• Suppresses expected errors
• Based on object type and error code
• Example: ORA-955 during CREATE TABLE ("table or view does exist")
• Enforces exclusive access for the upgrade process
• CLUSTER_DATABASE=FALSE
• No system triggers
• No AQ
• No resource manager
• No AWR
• etc.
ALTER SYSTEM SET _system_trig_enabled=FALSE SCOPE=MEMORY;
Autotune of undo retention is turned off.
ALTER SYSTEM SET _undo_autotune=FALSE SCOPE=MEMORY;
ALTER SYSTEM SET undo_retention=900 SCOPE=MEMORY;
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
ALTER SYSTEM SET enable_ddl_logging=FALSE SCOPE=MEMORY;
Resource Manager disabled during database migration: plan '' not set
ALTER SYSTEM SET resource_manager_plan='' SCOPE=MEMORY;
ALTER SYSTEM SET recyclebin=‘OFF’ DEFERRED SCOPE=MEMORY;
Resource Manager disabled during database migration
Note:This is an excerpt from the alert.log – these parameters will be set implicitly during a STARTUP UPGRADE

Why Upgrade is Different | Upgrade Mode
What happens during an upgrade?
• Upgrade scripts
• Example: c18.sql, i18.sql
• Install scripts
• Example: catalog.sql, catproc.sql, a18.sql

Parallel Upgrade | Non-CDB
17
A number of parallel processes
• Minimum 1
• Maximum 8
• Default 4
Non-CDB
$ dbupgrade -n 2

18
Phase 1
Phase 2
Phase 3
Phase n
Script 1 Script 2
Script 1 Script 2
Script 1
Script n
.
.
Worker 1
Script 3
Worker 2
Now idle

Does not scale linear
19
Contention

Parallel Upgrade | Container Database
20
A number of processors are assigned
• Minimum 4
• Maximum unlimited
• Default CPU count
$ dbupgrade -n 4
CDB

21
Each PDB gets a number of parallel processes
• Minimum 1
• Maximum 8
• Default 2
$ dbupgrade -N 2
CDB

22
But - there is another limit
CDB
Total number of processors (n)
Processor per PDB (N)
PDBs upgraded
simultaneously
=

Parallel Upgrade | Single Tenant
23
$ dbupgrade -n 4 -N 2
CDB$ROOT
PDB$SEED PDB1

Non-CDB is always faster
than single tenant
Parallel Upgrade | Non-CDB vs. Single Tenant
24

Parallel Upgrade | Multitenant
25
$ dbupgrade -n 4 -N 2
CDB$ROOT
PDB$SEED PDB1
PDB2 PDB3
PDB4 PDB5
PDB6 PDB7

Scale by upgrading
more PDBs simultaneously
Parallel Upgrade | Multitenant
26

Parallel Upgrade | Unplug-plug Upgrade
27
PDB1
CDB$ROOT
PDB$SEED
Higher release

Unplug-plug always faster than
Non-CDB
Single Tenant
Multitenant
Parallel Upgrade | Unplug-plug
28

29
Database
Upgrade
Internals
Photo
by
Daniel
Cheung
on
Unsplash

Upgrade Internals | How does the upgrade process work?
30
Components

31

32

33

34

Upgrade Internals | Timing
35

Upgrade Internals | CDB Upgrade
36

Upgrade Internals | CDB Upgrade Timings
37

Photo
by
Nathan
Dumlao
on
Unsplash
I Need More Power
I can't hold her together, Captain!

Faster Upgrades | Statement
During upgrade CPU is a vital resource

Faster Upgrades | Overview
• OCI Bare Metal host
• 16 OPCUs
• 768 GB memory
• NVMe disks
• CDB with 52 PDBs
• CPU_COUNT = 32
• SGA_TARGET = 80G
• PGA_AGGREGATE_TARGET = 20G
• Many database components (17 in total)
• Upgrade from 12.1.0.2 to 19

Faster Upgrades | CPU Utilization
0
10
20
30
40
50
60
70
80
90
100
00:00:00
00:05:00
00:10:00
00:15:00
00:20:00
00:25:00
00:30:00
00:35:00
00:40:00
00:45:00
00:50:00
00:55:00
01:00:00
01:05:00
01:10:00
01:15:00
01:20:00
01:25:00
01:30:00
01:35:00
01:40:00
01:45:00
01:50:00
01:55:00
02:00:00
02:05:00
02:10:00
02:15:00
02:20:00
02:25:00
02:30:00
02:35:00
02:40:00
02:45:00
02:50:00
02:55:00
03:00:00
03:05:00
03:10:00
03:15:00
03:20:00
03:25:00
03:30:00
03:35:00
03:40:00
03:45:00
03:50:00
03:55:00
04:00:00
04:05:00
04:10:00
Ideal utilization
Total upgrade time: 4 hours 8 minutes

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
Preupgrade check and fixups
• Gather dictionary and fixed objects stats in advance (7 days)

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
0
20
40
60
80
100
00:00:10
00:06:10
00:12:10
00:18:10
00:24:10
00:30:10
00:36:10
00:42:10
00:48:10
00:54:10
Gathering stats in advance saves 12 minutes

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
Upgrade CDB$ROOT
• Remove components
• AutoUpgrade automatically assigns 8 parallel processes to CDB$ROOT upgrade

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
0
20
40
60
80
100
00:00:10
00:05:50
00:11:30
00:17:10
00:22:50
00:28:30
00:34:10
00:39:50
00:45:30
00:51:10
00:56:50
01:02:30
01:08:10
01:13:50
01:19:30
01:25:10
01:30:50
01:36:30
01:42:10
01:47:50
01:53:30
01:59:10
02:04:50
02:10:30
02:16:10
02:21:50
02:27:30
02:33:10
02:38:50
02:44:30
02:50:10
02:55:50
Removing all components
13 minutes faster
All components installed

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
Upgrade PDB$SEED and user PDBs
• Add more PDBs (catctl -n)
• Keep parallel processes per PDB at default (2)
• Remove components from PDBs

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
0
20
40
60
80
100
00:00:10
00:06:20
00:12:30
00:18:40
00:24:50
00:31:00
00:37:10
00:43:20
00:49:30
00:55:40
01:01:50
01:08:00
01:14:10
01:20:20
01:26:30
01:32:40
01:38:50
01:45:00
01:51:10
01:57:20
02:03:30
02:09:40
02:15:50
02:22:00
02:28:10
02:34:20
02:40:30
02:46:40
02:52:50
02:59:00
03:05:10
03:11:20
03:17:30
03:23:40
03:29:50
03:36:00
03:42:10
32 parallel processes
54 parallel processes
upg1.catctl_options=-n 54
26 minutes faster

0
20
40
60
80
100
00:00:10
00:05:50
00:11:30
00:17:10
00:22:50
00:28:30
00:34:10
00:39:50
00:45:30
00:51:10
00:56:50
01:02:30
01:08:10
01:13:50
01:19:30
01:25:10
01:30:50
01:36:30
01:42:10
01:47:50
01:53:30
01:59:10
02:04:50
02:10:30
02:16:10
02:21:50
02:27:30
02:33:10
02:38:50
02:44:30
02:50:10
02:55:50
0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
All components installed
Removing all components
48 minutes faster

0
20
40
60
80
100
00:00:00
00:06:10
00:12:20
00:18:30
00:24:40
00:30:50
00:37:00
00:43:10
00:49:20
00:55:30
01:01:40
01:07:50
01:14:00
01:20:10
01:26:20
01:32:30
01:38:40
01:44:50
01:51:00
01:57:10
02:03:20
02:09:30
02:15:40
02:21:50
02:28:00
02:34:10
02:40:20
02:46:30
02:52:40
02:58:50
03:05:00
03:11:10
03:17:20
03:23:30
03:29:40
03:35:50
03:42:00
03:48:10
03:54:20
04:00:30
04:06:40
Post upgrade checks and fixups
• Recompilation (utlrp) already highly parallelized
• Postpone timezone file upgrade

Faster Upgrades | Conclusion
• Gather stats in advance
• Allow more PDBs per cycle
• Remove components

Database Upgrade
How do you upgrade?
Photo
by
Joshua
Sortino
on
Unsplash
Photo
by
Christian
DeKnock
on
Unsplash

Your Key to Successful Database Upgrades
1 2 3 4
Download and install
Oracle 19c
Download and install
newest RU
Download and use
newest AutoUpgrade
eDelivery MOS: 2118136.2 MOS: 2485457.1
SPM, STA, RAT for
Performance Stability

Database Upgrade | Supported Releases
18c
12.2.0.1
12.1.0.2
11.2.0.4
Older
19c
19c
19c
19c
19c

Database Upgrade | Supported Releases
Database Upgrade Guide

Database Upgrade | Intermediate Upgrades
Older 19c
• Ideally - upgrade as far as possible in the intermediate upgrade
• But often there is a platform upgrades as well
• So - it depends

AutoUpgrade
The ONLY recommended way to
upgrade databases
Photo
by
Joshua
Sortino
on
Unsplash

57 Copyright © 2021, Oracle and/or its affiliates
AutoUpgrade | Facts
AutoUpgrade today:
• 5 execution modes
• 38 single parameters
• 15 internal customizable parameters
Photo
by
Felipe
Echeverri
on
Unsplash

Click to add image
Why are there so many options?
Because you asked for it ...
58
Photo
by
Brian
McGowan
on
Unsplash

java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar [options]
Pros
+ Advanced spfile management
+ Advanced network files support
+ Advanced Architecture change support
+ Automatic restoration (GRP)
+ Full POSIX support for RAC
+ Fully resumable
+ In-place Upgrade
+ Advanced pre/post fixups
+ Multiple databases/CDBs at a time
+ The fastest development cycle
Cons
- No GUI (yet)
AutoUpgrade | Pros and Cons
59

START 1. DOWNLOAD 2. CONFIG 3. DEPLOY SUCCESS
Upgrade | AutoUpgrade - As Easy As 1-2-3
Supported releases
• 11.2.0.4
• 12.1.0.2
• 12.2.0.1
• 18
• 19

Download from My Oracle Support ID 2485457.1
3 MB Java file

Simple text file
upg1.source_home=/u01/app/oracle/product/12.2.0.1
upg1.target_home=/u01/app/oracle/product/19
upg1.sid=CDB1

One command
Advanced monitoring and logging
$ java -jar autoupgrade.jar -config cdb1.cfg -mode deploy

Supported target releases
• 12.2.0.1
• 18
• 19
• 21
• Any future release

AutoUpgrade | Flow-Overview
After Upgrade
System Tasks Database Checks User-Defined Tasks
Upgrade Core
Database Upgrade
Before Upgrade
User-Defined Tasks System Checks Database Checks

AutoUpgrade | Checks Overview
66
Components
amd_exists
apex_manual_upgrade
awr_expired_snapshots
em_present
exf_rul_exists
javavm_status
ols_sys_move
…
Configurations
case_insensitive_auth
auto_login_wallet_required
cdb_only_support
compatible_grp
compatible_not_set
dictionary_stats
dir_symlinks_exist
enable_local_undo
invalid_objects_exist
max_string_size_on_db
…
Spatial
disk_space_for_recovery_area
db_block_size
default_resource_limit
flash_recovery_area_setup
min_archive_dest_size
tablespaces
tempts_notempfile
…
Misc
network_acl_priv
new_time_zones_exist
parameter_deprecated
pending_dst_session
underscore_events
…

AutoUpgrade | Flow-Overview
After Upgrade
System Tasks Database Checks User-Defined Tasks
Upgrade Core
Database Upgrade
Before Upgrade
User-Defined Tasks System Checks Database Checks
115+ Checks
14+ System
19+ Checks
10+ System

AutoUpgrade | Availability and Support
All architectures (CDB and non-CDB)
All supported operating systems
All editions (SE2, EE)
All types (single instance and RAC)
Upgrades from all supported versions:
• 11.2.0.4 and newer
Upgrades to version:
• 12.2.0.1 with Jan 2019 Update or newer
• 18.5.0 (includes Jan 2019 Update) or newer
• 19.3.0 or newer

Alternative Upgrade Options
Any other ways to upgrade databases?
Photo
by
Joshua
Sortino
on
Unsplash
Photo
by
Tim
Mossholder
on
Unsplash

Alternative Upgrade Options | Command Line
preupgrade.jar
• Download the newest from MOS Note: 884522.1
• You need to do everything by yourself
• Not supported anymore from 21c
• java -jar autoupgrade.jar -preupgrade "target_version=21"
dbupgrade
• Can be used without parameters
• Wrapper for catctl.pl with default settings

Alternative Upgrade Options | GUI
Database Upgrade Assistant
• Not our choice
• Not fully resumable
• See Blog Post

72
AutoUpgrade
Essentials
Photo
by
Daniel
Cheung
on
Unsplash

AutoUpgrade | Essentials
Download
Configure
Analyze
Check
Upgrade
Always download latest version from MOS

Download
Configure
Analyze
Check
Upgrade
Check your version
$ java -jar autoupgrade.jar –version
build.hash 04dd9f2
build.version 19.7.5
build.date 2020/02/11 15:28:49
build.max_target_version 19
build.type production
Compare to latest version on MOS

Download
Configure
Analyze
Check
Upgrade
Shortest possible config file version
upg1.sid=CDB1
Or, generate a sample config file
$ java -jar autoupgrade.jar -create_sample_file config
Created sample configuration file /home/oracle/sample_config.cfg
Pro tip: upg1 is a prefix that you decide.
Use it to define multiple databases

Download
Configure
Analyze
Check
Upgrade
Analyze your database
$ java -jar autoupgrade.jar -config CDB1.cfg -mode analyze
...
upg> Job 100 completed
Please check the summary report at:
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
Pro tip: Analyze is similar to
running preupgrade.jar

Download
Configure
Analyze
Check
Upgrade
==========================================
Autoupgrade Summary Report
==========================================
[Date] Tue Jan 12 10:26:19 CET 2021
[Number of Jobs] 1
==========================================
[Job ID] 100
==========================================
[DB Name] CDB1
[Version Before Upgrade] 12.2.0.1.0
[Version After Upgrade] 19.9.0.0.0
------------------------------------------
[Stage Name] PRECHECKS
[Status] SUCCESS
[Start Time] 2021-01-12 10:25:58
[Duration] 0:00:20
[Log Directory] /u01/app/oracle/upg/CDB1/100/prechecks
[Detail] /u01/app/oracle/upg/CDB1/100/prechecks/cdb1_preupgrade.log
Precheck passed and no manual intervention needed
------------------------------------------
Summary report - text

Download
Configure
Analyze
Check
Upgrade
Summary report - HTML

Download
Configure
Analyze
Check
Upgrade

Download
Configure
Analyze
Check
Upgrade
Preupgrade report comes in:
• HTML
• Text
• JSON

Download
Configure
Analyze
Check
Upgrade
Upgrade
$ java -jar autoupgrade.jar -config CDB1.cfg -mode deploy
Have a cup of coffee and wait, or ...

Download
Configure
Analyze
Check
Upgrade
All the details
upg> status -job 101
Progress
-----------------------------------
Start time: 20/11/24 13:38
Elapsed (min): 13
Last update: 2020-11-24T13:48:52.139
Stage: DBUPGRADE
Operation: EXECUTING
Status: RUNNING
Stage summary:
SETUP <1 min
GRP <1 min
PREUPGRADE <1 min
PRECHECKS <1 min
PREFIXUPS 8 min
DRAIN <1 min
DBUPGRADE 3 min (IN PROGRESS)
Job Logs Locations
-----------------------------------
Logs Base: /home/oracle/autoupg_default/CDB1/CDB1
Job logs: /home/oracle/autoupg_default/CDB1/CDB1/101
Stage logs: /home/oracle/autoupg_default/CDB1/CDB1/101/dbupgrade
TimeZone: /home/oracle/autoupg_default/CDB1/CDB1/temp

Download
Configure
Analyze
Check
Upgrade
Success
upg> Job 101 completed
------------------- Final Summary --------------------
Number of databases [ 1 ]
Jobs finished successfully [1]
Jobs failed [0]
Jobs pending [0]
------------- JOBS FINISHED SUCCESSFULLY -------------
Job 101 for CDB1
---- Drop GRP at your convenience once you consider it is no longer needed ----
Drop GRP from CDB1: drop restore point AUTOUPGRADE_9212_CDB1122010
And it includes:
• Recompilation (utlrp)
• Timezone file upgrade
• Postupgrade fixups
• ... and so much more

One-liner using environment variables
One-liner using config_values
export ORACLE_SID=CDB1
export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1
export ORACLE_TARGET_HOME=/u01/app/oracle/product/19
java -jar autoupgrade.jar -config_values -mode analyze
java -jar autoupgrade.jar
-config_values "sid=CDB1,source_home=/u01/app/oracle/product/12.2.0.1,target_home=/u01/app/oracle/product/19"
-mode analyze

AutoUpgrade Demo
Copyright © 2021 Oracle and/or its affiliates.
Photo
by
Adam
Muise
on
Unsplash
Photo
by
Daniel
Cheung
on
Unsplash
From zero to hero in ten minutes

AutoUpgrade | Demo - Download
88

AutoUpgrade | Demo – Identify Databases
89

AutoUpgrade | Demo – Pick Your Target Home
90

AutoUpgrade | Demo – Create Configuration File - 1
91

92

93
Database db11204 and cdb18x will be set to archive mode

94

AutoUpgrade | Demo – Run AutoUpgrade - 1
95

AutoUpgrade | Demo – Run AutoUpgrade - 2
96

97
AutoUpgrade
Advanced Options
Photo
by
Ciprian
Boiciuc
on
Unsplash

AutoUpgrade | Advanced Options
Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Upgrade one or many databases
upg1.sid=CDB1
upg1.sid=CDB1
upg2.sid=CDB2
upg3.sid=DB11204
upg4.sid=SALES
upgn.source_home=/u01/app/oracle/product/12.1.0.2
upgn.target_home=/u01/app/oracle/product/19
upgn.sid=HR
One
Many

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Upgrade only when upgrade_node matches hostname
• Database upg1 will only be upgraded when AutoUpgrade gets executed
on server test_server01.mycorp.net
upg1.upgrade_node=test_server01.mycorp.net
upg1.sid=CDB1
upg2.upgrade_node=prod_server01.mycorp.net
upg2.sid=CDB2

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Batch-update parameters
global.del_during_upgrade_pfile=/home/oracle/global_del_during.ora
global.add_during_upgrade_pfile=/home/oracle/global_add_during.ora
global.del_after_upgrade_pfile=/home/oracle/global_del_during.ora
global.add_after_upgrade_pfile=/home/oracle/global_add_after.ora
upg1.sid=CDB1
upg1.add_after_upgrade_pfile=/home/oracle/upg1_add_after.ora
upg2.sid=CDB2
upg2.add_after_upgrade_pfile=/home/oracle/upg2_add_after.ora
# Example: global_add_after.ora
deferred_segment_creation=false
_cursor_obsolete_threshold=1024
_sql_plan_directive_mgmt_control=0
_use_single_log_writer=true

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Shell script execution
• Permitted extension options:
• Unix shell (.sh)
• Microsoft Windows batch (.bat, .cmd)
• Microsoft Windows PowerShell (.ps1)
global.before_action=/database/scripts/run_this_before.sh Y
global.after_action=/database/scripts/run_this_after.sh
upg1.sid=CDB1
upg1.before_action=/database/scripts/run_this_on_UPG1_before.sh
upg1.after_action=/database/scripts/run_this_on_UPG1_after.sh
Pass return code
back to AutoUpgrade

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Guaranteed Restore Points
• Default behavior:
• AutoUpgrade creates GRP except for
- Standard Edition 2
- restoration=no
• GRP will be kept
• GRP needs to be removed manually except for
- drop_grp_after_upgrade=yes will only remove it when upgrade
completed successfully
upg1.sid=CDB1
upg1.restoration=no
upg2.sid=CDB2
upg2.drop_grp_after_upgrade=yes

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Underscore parameters and events
• Underscores and events will be kept
upg1.sid=CDB1
upg1.remove_underscore_parameters=yes

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Postpone recompilation
• Recompilation happens after the upgrade
upg1.sid=CDB1
upg1.run_utl_rp=no

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Skip time zone upgrade
• Time zone adjustment happens post upgrade
• Database will be restarted several times
• Important when you use "Downgrade" as fallback strategy as time zone can't
be downgraded
upg1.sid=CDB1
upg1.timezone_upg=no

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
CDB
Concurrent PDB upgrades: n / N
Non-CDB coming soon
upg1.catctl_options=-n 64 -N 8
-n Total number of parallel processes
(min 4, max unlimited, default CPU_COUNT)
-N Number of parallel processes per PDB
(min 1, max 8, default 2)

Many Databases
Different Servers
PFILE
Shell Scripts
Restore Point
Underscores
Recompilation
Time Zone
Parallel
Monitoring
Use Python SimpleHTTPServer
Then open your browser:
• http://127.0.0.1:8000/state.html
AutoUpgrade: Refresh Status Information Automatically
cd <au_global_log_dir>/cfgtoollogs/upgrade/auto
python –m SimpleHTTPServer 8888

AutoUpgrade | Advanced Options – Monitoring Demo
Watch on YouTube

AutoUpgrade | Gathering Fixed Objects Stats
After an upgrade, or after other database configuration changes, Oracle strongly
recommends that you regather fixed object statistics after you have run
representative workloads on Oracle Database.
"
Never run it right after upgrade
Database 19c Upgrade Guide, chapter 7

Ask yourself: Do you remember this?
If not, DBMS_SCHEDULER to the rescue

BEGIN
DBMS_SCHEDULER.CREATE_JOB (
job_name => '"SYS"."GATHER_FIXED_OBJECTS_STATS_ONE_TIME"',
job_type => 'PLSQL_BLOCK',
job_action => 'BEGIN DBMS_STATS.GATHER_FIXED_OBJECTS_STATS; END;',
start_date => SYSDATE+7,
auto_drop => TRUE,
comments => 'Gather fixed objects stats after upgrade - one time'
);
DBMS_SCHEDULER.ENABLE (
name => '"SYS"."GATHER_FIXED_OBJECTS_STATS_ONE_TIME"'
);
END;
/
1. Create a .sql script

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl
-n 4 -e
-C 'PDB$SEED'
-b sched_gfos -d /home/oracle/sched_gfos/ sched_gfos.sql
2. Create a .sh script
upg1.after_action=/home/oracle/sched_gfos/sched_gfos.sh
3. Execute .sh script after upgrade
Further information and non-CDB example in blog post

113
AutoUpgrade
Case by Case
Photo
by
Will
Porada
on
Unsplash

Unplug / Plug / Upgrade
Photo
by
Adam
Muise
on
Unsplash
Photo
by
Katarzyna
Pe
on
Unsplash

Upgrade a single PDB
• Faster
• More flexible
• Requires compatible target CDB
• Not compatible with Flashback Database
• Consider using Refreshable PDBs
• Copy data files (target_pdb_copy_option)
AutoUpgrade | Unplug-plug Upgrade
115

116
upg1.sid=CDB12102
upg1.target_cdb=CDB19
upg1.pdbs=pdb1
upg1.source_home=/u01/app/oracle/product/12201
Watch on YouTube

Upgrade several PDBs
Rename a PDB
Copy data files on plug-in
upg1.pdbs=pdb1,pdb2,pdb3
upg1.pdbs=pdb1
upg1.target_pdb_name.pdb1=sales
upg1.pdbs=pdb1
upg1.target_pdb_copy_option.pdb1=file_name_convert=('pdb1','sales')

Current limitations:
• Does not support Data Guard
• Does not support TDE Tablespace Encryption

119
https://dohdatabase.com/how-to-upgrade-a-single-pdb

1. Create refreshable PDB
2. Upgrade with dbupgrade only
3. Open
Upgrade | Unplug-plug Upgrade With Refreshable PDB
120
New
DB System
19c
Current
DB System
12.2
Refreshes
via redo

Upgrade | Unplug-plug Upgrade With Refreshable PDB
121
https://dohdatabase.com/2020/06/15/upgrading-in-the-cloud-vm-db-systems-12-2-0-1-pdb-to-19c/

AutoUpgrade with Data Guard
Photo
by
Adam
Muise
on
Unsplash

Upgrade with Data Guard | Concept
Primary database
Standby
database
Oracle Home 11.2.0.4 Oracle Home 11.2.0.4
Oracle Home 19c Oracle Home 19c
Install new Oracle Home
Remember use latest
Release Update
Redo Apply
Stop redo apply
and shut down
Upgrade database Mount standby
in new Oracle Home
Redo Apply
Upgrade via
redo apply

AutoUpgrade | With Data Guard
124
• Detected by AutoUpgrade automatically
• Works for broker-managed and manual Data
Guard environments
• Primary database handled by AutoUpgrade
Standby database handled manually
Watch on YouTube

AutoUpgrade | With Data Guard
125
MAA Approach
• Keep standby online during upgrade
• Allows for faster go-live after upgrade
• Move standby database to new Oracle Home before upgrade
• Keep redo transport and redo apply on during upgrade

AutoUpgrade to a New Server
Photo
by
Hello
I'm
Nik
🎞
on
Unsplash

AutoUpgrade | Upgrade to a new server
Concept
1. Run the -mode analyze step on source server
2. Run the -mode fixups step on source server
3. Shutdown and restore your database plus config files to the target server
• This not done by AutoUpgrade
4. Start your database in STARTUP UPGRADE mode on target server
5. Run the -mode upgrade step on target server
Oracle AutoUpgrade between two servers

Source Server
• -mode analyze
• -mode fixups
• shutdown immediate
Target Server
• Update /etc/oratab
• Prepare ?/network/admin files
• STARTUP UPGRADE
• -mode upgrade
AutoUpgrade | Upgrade to a new server
128
upg1.sid=DB12
Copy database including redo logs, control files, SPFILE, password file
upg1.source_home=/tmp
upg1.sid=DB12

Plugin-Only with AutoUpgrade
Photo
by
Danilo
Alvesd
on
Unsplash

AutoUpgrade | Plugin AFTER Upgrade
Plug in non-CDB into an existing CDB without upgrading
• Non-CDB is upgraded already
• Config file
• AutoUpgrade
• Oracle AutoUpgrade between two servers – and Plugin?
upg1.sid=DB12
upg1.target_cdb=CDB2
java -jar autoupgrade.jar -config DB12NEW.cfg -mode deploy

What if ...
Tips and Tricks and Workarounds
Photo
by
Chris
Briggs
on
Unsplash

AutoUpgrade | What if ... your session is lost
132
• AutoUpgrade is fully resumable
• Restart using the same command line
• Previous work is preserved
- upgrade restarts from where it left
Watch on YouTube

AutoUpgrade | What if ... your session is lost
133
Or better, use nohup
Or even better, use a terminal multiplexer
• tmux
• screen
nohup java -jar autoupgrade.jar -config db.cfg -mode deploy -noconsole &

AutoUpgrade | What if ... a fixup fails?
To override a fixup
1. ANALYZE
2. Edit checklist
3. Edit config file
4. Deploy
java -jar autoupgrade.jar –config DB.cfg -mode analyze
Run analyze
It will produce a checklist:
../prechecks/<sid>_checklist.cfg

To override a fixup
1. Analyze
2. EDIT CHECKLIST
3. Edit config file
4. Deploy
----------------------------------------------------
[checkname] DICTIONARY_STATS
[stage] PRECHECKS
[fixup_available] YES
[runfix] YES
[severity] RECOMMEND
----------------------------------------------------
Edit checklist and find the failing fixup
----------------------------------------------------
[checkname] DICTIONARY_STATS
[stage] PRECHECKS
[fixup_available] YES
[runfix] NO
[severity] RECOMMEND
----------------------------------------------------
Pro tip: Set runfix to _SKIP to
prevent the check and fixup from
running

To override a fixup
1. Analyze
2. Edit checklist
3. EDIT CONFIG FILE
4. Deploy
Edit config file and specify your customer checklist
upg1.checklist=../prechecks/<sid>_checklist.cfg

To override a fixup
1. Analyze
2. Edit checklist
3. Edit config file
4. DEPLOY
Upgrade
java -jar autoupgrade.jar –config DB.cfg -mode deploy

AutoUpgrade | What if ... you need to restart?
Use AutoUpgrade to:
• Flashback the database
• Revert a plug-in operation (only when data files are copied)
• Revert a non-CDB to PDB conversion (only when data files are copied)
java -jar autoupgrade.jar -restore –jobs n
If you revert or restore in any other way, you must tell AutoUpgrade

If you revert or restore in any other way, you must tell AutoUpgrade
1. Clear recovery data for a specific job
2. Clear all recovery data
3. Remove AutoUpgrade log directories - use with caution
java -jar autoupgrade.jar -config DB.cfg -clear_recovery_data –job n
java -jar autoupgrade.jar -config DB.cfg -clear_recovery_data
global.autoupg_log_dir=/u01/app/oracle/cfgtoollogs/autoupgrade
upg1.log_dir=/u01/app/oracle/admin/DB1/upglogs
rm -rf /u01/app/oracle/cfgtoollogs/autoupgrade
rm -rf /u01/app/oracle/admin/DB1/upglogs

https://dohdatabase.com/is-autoupgrade-resumable/

What works
• Upgrade non-CDB and CDB with
auto-login keystore
What doesn't work
• Non-CDB to PDB conversion
• PDB plug-in
• Unplug-plug upgrade
AutoUpgrade | What if ... you have TDE?

AutoUpgrade | What if ... you have TDE?
What is in the pipeline
• Support for Keystore External Password Store
• Support for typing keystore password in AutoUpgrade console
• Proper Data Guard and RAC support

AutoUpgrade | What if ...
1. ZIP the entire global log directory:
java -jar autoupgrade.jar -config config.cfg -zip
2. Add the alert.log
3. Upload it to MOS

When should you change COMPATIBLE?
• 7-10 days after the upgrade
Which value?
• 11.2.0
• 12.1.0
• 12.2.0
• 18.0.0
• 19.0.0
Change COMPATIBLE for RUs?
• NEVER!
• Except for ...
• Flashback to GRP
• Downgrade
144

COMPATIBLE won't get changed by any upgrade
Use a shell script: AutoUpgrade and the COMPATIBLE Parameter
upg1.drop_grp_after_upgrade=YES
upg1.after_action=/home/oracle/scripts/change_compatible.sh
#!/bin/bash
# Shell script change_compatible.sh
. db19
sqlplus /nolog @/home/oracle/scripts/change_compatible.sql
connect / as sysdba
alter system set compatible='19.0.0' scope=spfile sid='*';
shutdown immediate
startup
exit
AutoUpgrade
Config
File
Shell
Script
SQL
Script
COMPATIBLE
Change

146
Database
AutoUpgrade
Diagnosability
Photo
by
James
Pond
on
Unsplash

Health Check | hcheck.sql
If your database is highly important, do a health check
• hcheck.sql - Script to Check for Known Problems (MOS Note: 136697.1)
147

AutoUpgrade | Job Diagnose
• Commands toolbox
upg>
lsj // List jobs
resume -job <number> // Restarts a job
status -job <number> // Show job status
restore -job <number> // Restores database from GRP
abort -job <number> // Aborts the specified job
148

Understanding the
directory structure
AutoUpgrade

AutoUpgrade | Understanding the directory structure
One job
Second job
150

AutoUpgrade | Log File Structure
• /cfgtoollogs
• ./upgrade/auto
• /database_1
• ./job_number
• ./prechecks
• ./preupgrade
• ./prefixups
• ./drain
• ./dbupgrade
• ./postupgrade
• ./temp
• /database_2
• …
General Logs + State Files
HTML Report
Upgrade Logs
AU Temp files
151

AutoUpgrade | Different Stages
• ANALYZE
[PRECHECKS]
• FIXUPS
[PRECHECKS,PREFIXUPS]
• DEPLOY
[GRP, PREUPGRADE, PRECHECKS, PREFIXUPS, DRAIN, DBUPGRADE, POSTCHECKS, POSTFIXUPS, POSTUPGRADE, NONCDBTOPDB, SYSUPDATES]
• UPGRADE
[DBUPGRADE,POSTCHECKS,POSTFIXUPS,SYSUPDATES]
• POSTFIXUPS
[POSTFIXUPS]
152

• /cfgtoollogs
./upgrade/auto
• /database_1
./job_number
./prechecks
./preupgrade
./prefixups
./drain
./dbupgrade
./postupgrade
./temp
• /database_2
./job_number
…
Depends on the execution mode
153

analyze
154

analyze => fixups
155

analyze => fixups => deploy
156

analyze => fixups => deploy
157

../prechecks
AutoUpgrade | Pre-Checks Folder [cdb18x] analyze[100] > fixups[102] > deploy[104]
158

../prechecks
AutoUpgrade | Pre-Checks Folder
cdb18x_preupgrade.html
cdb18x_preupgrade.log
cdb18x_checklist.cfg
Reports
Customize fixups execution
159

cdb18x_checklist.cfg
AutoUpgrade | Pre-Checks Folder
160

AutoUpgrade | dbupgrade Folder [cdb18x] analyze[100] > fixups[102] > deploy[104]
Upgrade logs

AutoUpgrade | General Progress Folder
General Log Files
AutoUpgrade
Control Files
State Files
Status Files

General Log Files
Status Files

General Log Files
AutoUpgrade
Control Files
State Files
Status Files

AutoUpgrade | temp Directory
• Before/During/After AutoUpgrade-Generated pfiles
• Recompilation scripts
• Restart scripts
• Network Files backups
• tnsnames.ora
• listener.ora
• sqlnet.ora
• TimeZone Upg Logs
• Context Files

How do we know if an issue occurred?
AutoUpgrade console | status files | log files
AutoUpgrade

AutoUpgrade | Console

AutoUpgrade | Status [status.json & progress.json]

AutoUpgrade | Log files

Stage where it failed

AutoUpgrade | If something bad happens?
Photo
by
David
Schultz
on
Unsplash

173
System Restore
Photo
by
Jon
Tyson
on
Unsplash
-config config.cfg
-restore -jobs 104

174
Contact Support
Photo
by
Jphilipp
Berg
on
Unsplash
-config config.cfg
-zip

https://MikeDietrichDE.com https://DOHdatabase.com https://donhk.dev/
Blogs
175

Recorded Web Seminars
https://mikedietrichde.com/videos/
YouTube Channel:
https://www.youtube.com/c/Oracle
DatabaseUpgradesandMigrations
176
Photo
by
Jonathan
Velasquez
on
Unsplash

YouTube | Oracle Database Upgrades and Migrations
177
YouTube Channel

178
Photo
by
Daniel
Cheung
on
Unsplash
Thank you!

33616611930205162156 upgrade internals_19c

More Related Content

What's hot (20)

Similar to 33616611930205162156 upgrade internals_19c (20)

Recently uploaded (20)

33616611930205162156 upgrade internals_19c