TechEvent 2019: Oracle to PostgreSQL - a Travel Guide from Practice; Roland Stirnimann - Trivadis
0 likes291 views
Roland Stirnimann presented an agenda on Oracle to PostgreSQL migrations. He discussed differences between Oracle and PostgreSQL, as well as between PostgreSQL and EDB Postgres Advanced Server. He described two migration projects: one analyzing 800 Oracle schemas for migration feasibility, collecting data as CSV and generating HTML reports; the other performing a proof-of-concept migration to EDB Postgres Advanced Server for an aviation application using EDB's migration assessment and toolkit.
TechEvent 2019: Oracle to PostgreSQL - a Travel Guide from Practice; Roland Stirnimann - Trivadis
- 1. Trivadis Blog@rstirnimann_ch Oracle to PostgreSQL A Travel Guide from Practice Roland Stirnimann
- 2. Roland • 14 years Trivadis • Oracle HA, migration • DevOps with Ansible • PostgreSQL • Data platforms rstirnimann_ch
- 3. Agenda • Warm-Up - Some Differences to Oracle… • PostgreSQL versus EDB Postgres • Project Experiences • Oracle to PostgreSQL • Oracle to EDB PostgreSQL Advanced Server • Conclusion
- 4. Warm-Up – Some Differences to Oracle…
- 5. • Oracle enforces constraints per statement, PostgreSQL per row • Set constraint to DEFERRABLE in PostgreSQL for the same behavior • Possible error: cannot use a deferrable unique constraint for referenced table • Referenced columns by a foreign key must be a non-deferrable unique or primary key INSERT INTO demo VALUES (1),(2); UPDATE demo SET n=n+1; ERROR: duplicate key value violates unique constraint "demo_pk" DETAIL: Key (n)=(2) already exists. CONSTRAINT behavior (demo) Hint: INSERT of two rows at the same time does not work in Oracle. VALUES (1),(2) Source: Mathias Zarick, Trivadis Vienna
- 6. • Different behavior of SELECT INTO • Oracle: Restrictive – allows only one value • PostgreSQL: Tolerant – returns the first value of the SELECT Oracle CREATE OR REPLACE FUNCTION get_bal(acc_no IN NUMBER) RETURN NUMBER IS acc_bal NUMBER(11,2); BEGIN SELECT balance INTO acc_bal FROM accounts WHERE account_id = acc_no; RETURN acc_bal; END; / FUNCTION in PL/SQL and PL/pgSQL (demo) PostgreSQL CREATE OR REPLACE FUNCTION get_bal(acc_no IN INTEGER) RETURNS INTEGER AS $$ DECLARE acc_bal INTEGER; BEGIN SELECT balance INTO acc_bal FROM accounts WHERE account_id = acc_no; RETURN acc_bal; END; $$ LANGUAGE plpgsql; Source: Mathias Zarick, Trivadis Vienna
- 7. Statement error – ROLLBACK (demo) • Statement error in PostgreSQL • Uses by default AUTOCOMMIT • Rollback to the beginning or to the last save point • Transactional DDL support rollbacks DDL statements as well (e.g. CREATE TABLE) • Statement error in Oracle • Only the failed statement will be discarded • Transaction is still open Source: Mathias Zarick, Trivadis Vienna
- 8. What is crucial for the Migration? Test, Test, Test…
- 10. EDB Postgres platform Tools and support for enterprise level usage Source: https://www.enterprisedb.com/
- 11. EDB subscriptions based on number of CPU Source: https://www.enterprisedb.com/
- 12. EDB Postgres Advanced Server – the same core • EDB is based on PostgreSQL (binary compatible) • Additional enterprise features • Security: Password profiles, EDB*Wrap, etc. • Performance features: Hints, extended analysis, resource manager, etc. • Developer: Oracle PL/SQL, hierarchical query, synonyms, Oracle DBMS_* packages, etc. • Tools: BART, failover manager, PEM, migration toolkit, etc. • Oracle compatibility for simpler migration • EDB documentation extends the community documentation by the specific EDB features • https://www.postgresql.org/docs/11/index.html • https://www.enterprisedb.com/docs/en/11.0/EPAS_Guide_v11/toc.html • EDB has several well-known PostgreSQL developers on board
- 14. Project 1 - Energy supplier Estimation for the migration effort to PostgreSQL/EDB • Strategy: • 80% of all databases are in the AWS cloud until 2020 • Move away from commercial RDBMS and no vendor lock-in • Motivation: • Hardware renewal for Oracle RAC is not required • For the time being, the existing hardware is sufficient if some applications are moved to AWS (PostgreSQL) • Project goal: • Analysis tool for a migration feasibility assessment into the AWS cloud for the existing 800 Oracle schemas • Target-RDBMS is either PostgreSQL (RDS, DBaaS) or EDB Postgres AS (EC2, IaaS)
- 15. Project 1 - Requirements • No direct database access. Collecting and analyzing happens separately • Step 1: Collect data as CSV • Step 2: Analyze data and present them as HTML report • No software installation, only a script deployment is required • Gathering the required data: • gather_db_details.sql: Feature usage, Object types and schema details • gather_db_behavior.sql: Application behavior, e.g. programs, drivers accessing the application • gather_db_metrics.sql: Metrics about the resource usage (CPU, I/O) • Oracle RAC awareness • Important to assign the Oracle instances to the corresponding AWS DB-instance-classes https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html
- 16. • Execute the collector scripts in Oracle as DBA user • Option -M is only available since 12.2, otherwise use set colsep in SQL*Plus • Export the result from SQL Developer into a text file delimited by ; export NLS_DATE_FORMAT='dd.mm.yyyy hh24:mi:ss' sqlplus -S -M 'CSV on delimiter ;' system@REPO1 @../sql/gather_db_details.sql >gather_db_details.csv sqlplus -S -M 'CSV on delimiter ;' system@REPO1 @../sql/gather_db_behavior.sql >gather_db_behavior.csv sqlplus -S -M 'CSV on delimiter ;' system@REPO1 @../sql/gather_db_metrics.sql >gather_db_metrics.csv Project 1 - Collect information as CSV
- 17. • CSV files are used as input for the report creation perl -I ../lib mactl.pl --input-files "REPO1;../csv/gather_db_details.csv" --behavior-input-files "REPO1;../csv/gather_db_behavior.csv" --metric-input-files "REPO1;../csv/gather_db_metrics.csv" --output-directory ../html --schemas-only Project 1 - Analyze the collected data
- 18. Project 1 - Report demo
- 19. Project 1 - Next steps • Identify easily portable applications and migrate to PostgreSQL or EDB Postgres AS • After first experience try the harder ones • Migration tools for schema or database • PostgreSQL: ora2pg http://ora2pg.darold.net • EDB Postgres AS: Migration Toolkit https://www.enterprisedb.com/products/edb-postgres-platform/edb-migration-tool-kit • Get some Oracle compatibility in PostgreSQL with orafce tool https://github.com/orafce/orafce • EDB Postgres AS has comprehensive Oracle compatibility out-of-the-box
- 21. Project 2 - Aviation Migration to EDB Postgres Advanced Server • Strategy: • Consolidation of vendors • Analyzing cost saving potentials • Motivation: • Reducing maintenance and support costs • Complete renewal of an application (Oracle Forms) • Project goal: • Find out the feasibility of replacing Oracle with EDB Postgres Advanced Server • Knowledge about the migration scenarios and tools • Know the required structural changes in the code
- 22. Project 2 - Requirements • Perform a two-piece proof-of-concept • EDB: Database migration assessment (DMA) and online EDB Postgres migration portal • Trivadis: Practical migration part • Timely limited EDB Postgres Advanced Server license • Doing based on a complex and critical Oracle database • Customer provided a server with OS (Centos 7) • EDB Postgres Advanced Server 11 • Access to the Oracle database via SQLNet
- 23. Project 2 - EDB database migration assessment • Input was based on a structure export of the source database (export script from EDB) • PDF report from EDB Professional Services • It looks worse than it really is as we will see later during the practical part!
- 24. Project 2 - DMA classification • Classification of objects like procedures, functions, tables, indizes, etc. • The status Invalid occurs often because of dependencies to objects in other schemas • Incompatible means not impossible • Requires some additional effort in the code • Examples: Index organized table must be replaced or some reserved words must be enclosed by quotes
- 25. Project 2 - EDB Postgres migration portal (1) • Upload of the structure export incl. analysis (https://www.enterprisedb.com/edb-postgres- migration-portal)
- 26. Project 2 - EDB Postgres migration portal (2) • EDB offers many ready-to-use solutions in Postgres for most of the known incompatibilites
- 27. • EDB MTK is a CLI tool • Documentation: https://www.enterprisedb.com/docs/en/52.0/MTK_Guide_v52.0/toc.html • Installation via EDB YUM repository (yum install edb-migrationtoolkit) • Configuration file contains connection details to Postgres and Oracle vi /usr/edb/migrationtoolkit/etc/toolkit.properties SRC_DB_URL=jdbc:oracle:thin:@192.168.38.186:1521:DB01 SRC_DB_USER=system SRC_DB_PASSWORD=pw TARGET_DB_URL=jdbc:edb://localhost:5444/edb TARGET_DB_USER=enterprisedb TARGET_DB_PASSWORD=pw /usr/edb/migrationtoolkit/bin/runMTK.sh –help Project 2 - EDB migration toolkit (1)
- 28. Project 2 - EDB migration toolkit (2) • Online or offline migration • Online exports from Oracle and imports directly to EDB Postgres • Offline (-offlineMigration) exports into SQL files for manual import • Only structure and/or data • -schemaOnly export/import only the sturcture (DDL) • -dataOnly export/import only the data • Without explicit parameter definition it copies structure and data • Different options for importing of different objects types: -allTables, -allSequences, -skipFKConst, etc. • Oracle specific options: -allProfiles, -allDBLinks, -allSynonyms, etc. • Further migration options, e.g. to change data types during the migration
- 29. • Modify/correct DDL scripts within the export directory ~/mig • Create Postgres database user, schema and tablespace • Load the structure • Offline structure export from Oracle (select any dictionary privilege) runMTK.sh -offlineMigration ~/mig -schemaOnly -logDir ~/mig/logs -sourcedbtype oracle -targetSchema schema1 schema1 create user admin password 'xxx'; create database mydb with owner admin; c mydb enterprisedb create user schema1 with login identified by pw; create tablespace ts_schema1 owner schema1 location '/opt/tbs/schema1'; edb-psql -f mtk_schema1_ddl.sql -o mtk_schema1_ddl.log mydb schema1 Project 2 - MTK approach
- 30. • Many initial errors because of dependencies • Included further schemas • Created a database link to another database (Instant Client required) c mydb schema1 CREATE DATABASE LINK oradb.world CONNECT TO user1 IDENTIFIED BY 'pw' USING '//192.168.40.223:1521/ORADB’; select count(*) from table1@oradb.world; Project 2 - Findings (1) • Import order is key, order of schemas and DDL scripts • Permissions (GRANT) are required between schemas as well grant select on all tables in schema schema1 to schema2,schema3,schema4;
- 31. • ROWID functionality in EDB Postgres activated Parameter default_with_rowids=on in $PGDATA/postgresql.conf • Several syntax fixes required where the EDB parser is somehow more restrictive than Oracle • Few keywords as column names had to be enclosed in quotation marks • BLOB data type changed for one table to BYTEA • Granted access on SYS package UTL_FILE and a directory object created • search_path extended in DDL files with “public” for the visibility of public synonyms • Tablespace names defined in DDL files to create the objects correctly SET search_path=schema1,public; SET default_tablespace = ts_schema1; Project 2 - Findings (2)
- 32. Project 2 - Recommendations • Having the right persons at the table (DBA and developer)! • Work in parallel on application and database related issues • Agile, interactive approaching - migrate, verify, correct, rollback and the same again • Do not try to fix all potential problems at the first run • Start with the structure because loading is fast but the potential for issues is higher • Use offline migration to modify scripts before importing them • Data migration should be well thought out due to the long run time • Identify large objects and migrate them separately without MTK (delta-migration) • Basically we expect less issues during data migration • BUT: The run time can be bad especially with the standard JDBC copy (downtime) • Search for alternatives like database link (parallelism)
- 33. Project 2 - Next steps • Complete the structure migration including all dependencies • Feasibility of the data migration • Accepted downtime defines the maximum of run time for the go-live • Based on that the migration concept has to be created (database link, etc.) • Test very well the connectivity of interfaces to Postgres (drivers) • Create a Postgres operation concept • The 15 days for porting the structure alone is not enough!
- 34. Conclusion
- 35. Conclusion • Many things are technically possible with corresponding effort • EDB Postgres simplifies the migration of Oracle features heavily (PL/SQL) • Which path does the further development of ported applications follow, Oracle or Postgres? • Postgres as alternative RDMBS in the company is a wise decision • EDB offers support for community PostgreSQL as well • Oracle compatibility is less important for new project without an Oracle history • Knowledge level about PostgreSQL varies heavily in companies (training)