Trivadis TechEvent 2017 Oracle to My SQL Migration - Challenges by Robert Bialek and Robert Kruzynski

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Oracle to MySQL Migration – Challenges
Robert Bialek
Principal Consultant
Robert Kruzynski
Principal Consultant

Who Am I
Principal Consultant and Trainer at Trivadis GmbH in Munich.
– MSc in Computer Engineering.
– At Trivadis since 2004.
Focus:
– Oracle Database High Availability.
– Database Architecture/Internals.
– Backup/Recovery.
– Troubleshooting/Performance Tuning.
– Linux.
– Trainer: O-GRINF, O-RAC, O-DG.
15.07.20172 Oracle to MySQL Migration – Challenges

Who Am I
Principal Consultant and Trainer at Trivadis GmbH in Munich.
– Dipl.-Ing. Technische Informatik (TU Berlin)
– At Trivadis since 2002
Focus:
– Troubleshooting/Performance Tuning.
– Capacity and Resource Management.
– Automatization/Optimization.
– Backup/Recovery.
– Tool Owner: TVD-CapMan, TIB (Trivadis Intelligent Backup)
– Trainer: O-TUN, O-TUN-PRACT.
15.07.20173 Oracle to MySQL Migration – Challenges

Customer-Driven Trend
Two strategic Trivadis IMS customers switched the „default“ database engine from
Oracle to MySQL for new projects/applications.
Target environment:
– MySQL Enterprise Edition (optionally with data replication).
– Red Hat Enterprise Virtualization (Live Migration).
– Two security classes: standard and confidential (secret as of now not available).
– Goal: automatic full software stack provisioning.
In 2017, one of the customers decided to start a Proof-of-Concept project for
migration of exising applications (fixed-price).
15.07.2017 Oracle to MySQL Migration – Challenges4

Customer Project Goals
Which and how many Oracle applications/schemas can be migrated to MySQL?
What are the technical differences and challenges?
Categorize the existing Oracle databases (~1700) into migration (difficulty) classes.
– Should be performed automatically and based on customer thresholds, which can
be changed at any time.
Migration method should be successfully demonstrated on 5 Oracle applications.

Our Challenges
Create Oracle2MySQL migration matrix (features, data types, ...).
Define database migration (difficulty) classes.
Create a framework to scan the whole Oracle database environment and categorize
schemas/databases into migration (difficulty) classes.
Generate MySQL schema DDLs.
Migrate data from Oracle to MySQL database and check data correctness after
migration.
There is no framework on the market which supports all, or at least most of the
requirements.
– What we need is a small, but yet powerful framework.
– This way, a new tool was born: TVD-Espresso ☺.

Agenda
Oracle to MySQL Migration – Challenges15.07.2017
1. Oracle Database/Schema Classification
2. MySQL Schema Generator
3. Data Migration
4. Summary
7

Oracle Database/Schema
Classification

Migration Classes – Overview
Database/Schema classification based on features, data types, usage patterns, ...
New classes can be easily introduced.
15.07.2017
Migration Class Description
Easy (1) Automatic migration is possible.
Intermediate (2) Migration ist possible, requires some manual tasks
and/or re-coding.
Difficult (3) Migration ist possible, requires many and complex
manual tasks and/or re-coding.
Very Difficult (4) Migration ist not recommended (technical and/or
financial reasons).
Oracle to MySQL Migration – Challenges9

Example Migration Classes (1)
Class – Easy
Heap Table
Index-Organized Table
B-Tree Index
PK,FK,UK,NOT NULL Constraints
DATA_TYPE_CATEGORY_1:
▪ LOB (<4GB), (VAR)CHAR, LONG,
RAW, DATE, TIMESTAMP (<7),
NUMBER, ...
Class – Intermediate
Bitmap Index
View
Oracle Outer-Join Syntax
Synonym
Database Job

Class – Difficult
Temporary Table
Function-Based Index
Materialized View
Maximum index key length 3072
bytes
CHAR data type length > 255
Class – Intermediate/Difficult
Check Constraints:
▪ >0 Intermediate <=20
▪ >20 Difficult
Sequences:
▪ >0 Intermediate <=10
▪ >10 Difficult

Class – Very Difficult
Advanced Queueing
PL/SQL Package
Database Link
TDE, Encrypted columns
TIMESTAMP WITH [LOCAL]
TIMEZONE
Class – Difficult/Very Difficult
Procedures, Functions, Triggers:
▪ >0 Difficult <=10
▪ >10 Very Difficult
Object Source Code Lines:
▪ >0 Difficult <=100
▪ >100 Very Difficult

Migration Classes – Configuration
Metadata is stored in Oracle table.
SQL> SELECT FEATURE_NAME, DIFFICULTY "DIF.", MAXIMUM_COUNT MAXC, MAXIMUM_LENGTH MAXL,
DIFFICULTY_HIGH DIFH, MAXIMUM_COUNT_HIGH MAXH, DESCRIPTION
FROM MIGSCAN.MIGRATION_FEATURE;
FEATURE_NAME DIF. MAXC MAXL DIFH MAXH DESCRIPTION
------------------------------ ---- ---- ---- ---- ---- ----------------------------------------------
DATA_TYPE BFILE 4 Does not exist in MySQL
DATA_TYPE CATEGORY_NOT_1 2 0 Simple data types
DATA_TYPE CHAR_TOO_LONG 3 0 CHAR is restricted to 255 bytes in MySQL
DATA_TYPE TIMESTAMP_WITH_TIMEZONE 4 Does not exist in MySQL
MAX INDEX KEY LENGTH 3 3072 Max index key length is hard limited
OBJECT CONSTRAINT CHECK 2 0 3 20 Does not exist in MySQL
OBJECT CREDENTIAL 4 Does not exist in MySQL
OBJECT DATABASE_LINK 4 Does not exist in MySQL
OBJECT ENCRYPTION_COLUMN 4 0 Binlogs are not encrypted in MySQL
OBJECT INDEX FUNCTION_BASED 3 0 Does not exist in MySQL
OBJECT SYNONYM 2 Does not exist in MySQL
OBJECT SOURCE_CODE_LINES 3 0 4 100 Require re-engineering
SYS GRANTS 1 Application re-coding could be necessary

Database/Schema Classification (1)
Job scheduled from a central location:
– Implemented in shell, PL/SQL, SQL.
15.07.2017
DB-Server List
global_mig_scan.sh
target_sql_all_dbs.sh
target_mig_scan.sql
target_mig_scan.sql
target_mig_scan.sql
target_mig_scan.sqlAnalysis
Repository-DB
- Meta Information
- Scan Results
SQL
SQL
SQL
SQL
SQL, SQL, ...
ETL

Database/Schema Classification (2)
What the scan job cannot automatically find out:
– Application usage of specific SQL functions.
– Usage of dictionary objects.
– Usage of anonymous PL/SQL code.
Some information is extracted from V$ views, but this might be not reliable.

Database/Schema Classification Results (1)
Example classification results
– Fine tuning still necessary!
15.07.2017
6%
22%
20%
52%
Cat. 1
Cat. 2
Cat. 3
Cat. 4
2% 2%
15%
81%
Cat. 1
Cat. 2
Cat. 3
Cat. 4
MAX_DIFFICULTY_DB [%]
MAX_DIFFICULTY_SCHEMA [%]

Database/Schema Classification Results (2)
Classification Results are stored by ETL job in Oracle table.
SQL> SELECT MAX_DIFFICULTY_DB mdb, MAX_DIFFICULTY_SCHEMA msch, SCHEMA_NAME, DB_SCHEMAS, HEAP_TABLES,
DIFFICULT_FEATURES
FROM MIGSCAN.MIGRATION_REPORT;
MDB MSCH SCHEMA_NAME DB_SCHEMAS HEAP_TABLES DIFFICULT_FEATURES
---- ---- -------------------- ---------- ----------- ------------------------------------------------
3 1 A4MBBBPRSPOI 5 1
3 3 A4MBBBDWA1 5 5 OBJECT SEQUENCE=3,OBJECT SOURCE_CODE_LINES=18,
OBJECT TRIGGER=3,
3 3 A4MBBBMVS 5 7 OBJECT SEQUENCE=2,
3 3 A4MBBBMVSGEO 5 7 OBJECT SEQUENCE=2,
3 3 A4MBBBONUP 5 87 MAX INDEX KEY LENGTH,OBJECT FUNCTION=1,
OBJECT PROCEDURE=1,OBJECT SEQUENCE=3,
OBJECT SOURCE_CODE_LINES=39,
4 1 A4MBBAONUP 87 8
4 1 A4MBBBMOBKEY2 55 19
4 1 A4MBBBPRSPOI 7 1
4 3 A4MBBBDESTIM 87 31 OBJECT FUNCTION=1,OBJECT SEQUENCE=1,
4 3 A4MBBBDESTIM 89 31 OBJECT PROCEDURE=1,OBJECT SEQUENCE=1,

MySQL Schema Generator

MySQL Schema Generator – Overview
Generates in the source environment MySQL schema DDLs:
– Tables, Indexes, Constraints.
Responsible for data type mapping.
15.07.2017
SET SERVEROUTPUT ON
BEGIN
oracle2mysql.gen_tab_ddl(
in_schema => '<APP_NAME>',
in_cascade => TRUE,
in_file => '<APP_NAME>_all.sql');
END;
/
Gen. DDL
MySQL
DDL
Oracle
mysql> source <APP_NAME>_all.sql

MySQL Schema Generator – Data Mapping
15.07.2017
Oracle MySQL
VARCHAR2,NVARCHAR2
• <4001 Bytes
• >4000 Bytes
• VARCHAR
• TEXT
CHAR,NCHAR
• <256
• >255
• CHAR (different padding behaviour)
• Needs clarification.
LONG (max. 2GB) LONGTEXT (max. 4GB)
LONG RAW (max. 2GB) LONGBLOB (max. 4GB)
CLOB,NCLOB (max. 8 - 128 TB) LONGTEXT (max. 4GB)

15.07.2017
Oracle MySQL
BLOB (max. 8 - 128 TB) LONGBLOB (max. 4GB)
RAW (max. 2000 bytes) VARBINARY (4000 bytes)
TIMESTAMP
• max. PRECISION 6
• PRECISION 7-9
• TIMESTAMP
• <NOT_SUPPORTED>
DATE DATETIME
NUMBER(precision,scale) DECIMAL(precision,scale)
NUMBER(precision) DECIMAL(precision)
NUMBER(precision,-scale) <NOT_SUPPORTED>

15.07.2017
Oracle MySQL
NUMBER (*) • Default: DECIMAL(65,30)
BINARY_FLOAT DOUBLE
BINARY_DOUBLE DOUBLE
FLOAT DOUBLE
DEFAULT (Constant-Value, Functions) DEFAULT (Constant-Value, Functions*)

Data Migration

Data Migration – Overview
Data migration performed by TVD-Espresso using JDBC-Thin API:
– Automatic data type migration and conversion (including LOBs, LONGs and RAW).
– Makes sure that all migrated rows are identical (column values in every row).
– Migrates all or selected tables in a schema, optionally in parallel.
15.07.2017
java -classpath ojdbc6.jar:mysql-connector-java-5.1.42-bin.jar:.
com.trivadis.espresso.Espresso
--truncate
--tablepattern=%
--noflashback
--parallel=10
--sourceowner=<SCHEMA>
--sourcejdbc=jdbc:oracle:thin:@//<SERVER>:<PORT>/<SERVICE_NAME>
--sourceuser=<MIGRATION_USER>
--sourcepass=xxxxxxxxxxxxxxx
--destjdbc=jdbc:mysql://<SERVER>:<PORT>/<SCHEMA>
--destuser=<MYSQL_USER>
--destpass=xxxxxxxxxxxxxxx

Data Migration – Example Output
Example data migration output
15.07.2017
INFO: loading oracle driver
INFO: loading mysql driver
INFO: connecting to jdbc:oracle:thin:@//<SERVER>:<PORT>/<SERVICE_NAME> user mysqlmig
INFO: connecting to jdbc:mysql://<SERVER>:<PORT>/<SCHEMA>...
INFO: copying table
CEIDMAP_Z6:..........................................................................
.................................................... 12664975 rows processed in
466617 ms
INFO: row count of table CEIDMAP_Z6 is 12664975 OK
INFO: comparing column values in table CEIDMAP_Z6: finished in 299267 ms
CEIDMAP_Z6.SAPDOCID CHAR OK
CEIDMAP_Z6.ARCDOCID CHAR OK
CEIDMAP_Z6.DOCPROT NUMBER OK
CEIDMAP_Z6.NOTE NUMBER OK
CEIDMAP_Z6.STATE NUMBER OK

Data Migration – Example Output
Summary
– Performance („SELECT * FROM table ORDER BY PK”)
15.07.2017
EXECUTOR STATS: started worker threads 66
EXECUTOR STATS: finished worker threads 66
EXECUTOR STATS: copied tables 66
EXECUTOR STATS: copy exceptions 0
EXECUTOR STATS: rows copied 109376002
EXECUTOR STATS: compare exceptions 0
EXECUTOR STATS: select-order-by time source 201 s
EXECUTOR STATS: select-order-by time destination 1048 s

Summary

Summary (1)
With the newly developed framework/solution, we are able to fulfill all the customer
project requirements/goals.
– Find the best migration candidates.
– Perform the migration including data quality check.
– Support re-engeneering tasks.
With our experience and the tools we can support similar projects in any environment
and any other DBMS.
– JDBC connection to both systems is required.

Summary (2)
Does it make sense to migrate existing applications between different DBMS
products?
– It depends…
It might be the only choice for systems using cloud infrastructure.
Necessary changes on application level should not be underestimated! This part
might kill many migration projects ☺.
There could be also a lot of potential to optimize the usage of existing environments
(better resource management, tuning, software editions, ...).

Session Feedback – now
30 15.07.2017
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as "svv")
– Password: sent by e-mail...
Oracle to MySQL Migration – Challenges

Trivadis TechEvent 2017 Oracle to My SQL Migration - Challenges by Robert Bialek and Robert Kruzynski

More Related Content

What's hot (20)

Similar to Trivadis TechEvent 2017 Oracle to My SQL Migration - Challenges by Robert Bialek and Robert Kruzynski (20)

More from Trivadis (20)

Recently uploaded (20)

Trivadis TechEvent 2017 Oracle to My SQL Migration - Challenges by Robert Bialek and Robert Kruzynski