Simplifying EBS 12.2 ADOP - Collaborate 2019
1 like1,492 views
Viscosity is a consulting firm specializing in Oracle technologies. Alfredo Krieg, a senior cloud architect at Viscosity, discussed ADOP concepts including the prepare, apply, finalize and cutover phases. He covered best practices like keeping systems up-to-date with ETCC and using delta synchronization. Potential issues like locking during prepare and apply were also discussed along with recommendations like shutting down third-party apps before cutover.
![ADOP Concepts - Prepare
• Configuration changes
– If any configuration changes detected
• Clone configuration from run to patch
• Execute autoconfig on patch fs
09:46:46 00:00:00 ad.bin.adop EVENT Detecting configuration
changes.
09:46:46 00:00:00 ad.bin.adop PROCEDURE [START] Looking for file
system configuration changes
09:46:46 00:00:00 ad.bin.adop EVENT. File system configuration
changes detected.](https://image.slidesharecdn.com/simplifyingebs12-190407192804/85/Simplifying-EBS-12-2-ADOP-Collaborate-2019-24-320.jpg)
![Blocking locks and Prepare
Error on session xx:
Creating database patch edition.
Log: @ADZDSHOWLOG.sql
Failed to execute SQL statement:
declare
result varchar2(10);
begin
ad_zd.PREPARE('Y’);
…
Error Message: ORA-20001: Error while calling ad_zd.prepare.ORA-04021:
timeout occurred while waiting to lock object
ORA-06512: at line 9 (DBD ERROR: OCIStmtExecute) [UNEXPECTED]Error creating
patch edition.
[UNEXPECTED]Error calling runCheckAndCreatePatchEdition subroutine.
[ERROR] Prepare phase has failed.](https://image.slidesharecdn.com/simplifyingebs12-190407192804/85/Simplifying-EBS-12-2-ADOP-Collaborate-2019-41-320.jpg)
Simplifying EBS 12.2 ADOP - Collaborate 2019
- 1. Session ID: Prepared by: Remember to complete your evaluation for this session within the app! 10755 Simplifying EBS 12.2 ADOP 1/12/2019 Alfredo Krieg Senior Cloud Performance Architect Viscosity North America @alfredokrieg
- 3. @ViscosityNAviscosityna.com Leverage what Makes Viscosity Different Our Mission: •We Enable Business Transformation at a Time when Companies must Change to Survive
- 4. @ViscosityNAviscosityna.com Viscosity is a team of experts specializing in Oracle Cloud and Engineered Systems. We are lot more than just Oracle SharePlex Platinum Partner MongoDB Consulting Partner Neo4J Consulting Partner Kyriba - Treasury Management
- 5. We’ve written 22 books on Data, Cloud, and Oracle… Industry Experts o Viscosity founders hold 27+ years each in the Oracle space o Authors of 22 books in the Oracle space o 5 ACE Directors, only 36 in the United States. 1 Oracle ACE o Direct connections to Oracle support and Product Managers support o Expertise in Engineered Systems, ZFS, ODA, Oracle 11g, 12c, 18c, RAC/ASM, Data Guard, Performance Tuning & Oracle Licensing o Hire only best of the best
- 6. @ViscosityNAviscosityna.com 6 Viscosity Pillars And Delivery Models Database InfrastructureAppsData Data Integration IAAS / PAAS Data Warehousing Analytics GoldenGate SAAS / PAAS CX ERP SCM HCM PAAS Web Applications Mobility EBS / JDE / PS IAAS Public Cloud Bare Metal Cloud AWS Azure Bluemix Engineered Systems Exa- ZFS @Customer X86 / Sparc Workshops Assessments Proof of Concepts Training Turnkey Projects Managed Services
- 7. @ViscosityNAviscosityna.com Performance Health Checks How’s it running? Apps SaaS/PaaS, Mobility, Application Development DBA Services Remote and On-site On-Call Support Managed Services ZERO DOWNTIME Migrations Professional Services Where you need it most Staff Aug Workforce Capacity on Demand Oracle License Management Get the most out of your Oracle investment 7
- 12. Alfredo Krieg • Oracle ACE • Senior Cloud Performance Architect at Viscosity North America • Database Performance Tuning, Exadata, OEM and EBS • Oracle Technologies since 2004 • Blog www.alfredokriegdba.com • alfredo.krieg@viscosityna.com @alfredokrieg
- 13. Disclaimer • The opinions expressed in this presentation are solely mine, and not that of my employer.
- 14. Agenda • ADOP concepts – ADOP phases • Best Practices • ETCC • ADOP and DB Patches • Issues and troubleshooting • Final Recommendations
- 15. ADOP – How did I get here? • Upgrade to 12.2 from 11i, 12.0 or 12.1 • Fresh install of 12.2 Photo by Caleb Jones on Unsplash
- 16. ADOP Concepts • ADOP stands for AD Online Patching • Used to apply patches to EBS 12.2 (adpatch prior releases) • Capable of storing multiple application editions in the DB (EBR) • Dual application tier file system (fs1 & fs2 “or” run & patch) • 4 new environment variables – $FILE_EDITION – $RUN_BASE – $PATCH_BASE – $NE_BASE
- 17. ADOP Concepts • ADOP stages – Prepare – Apply – Finalize – Cutover – Cleanup • Special – Abort – Fs_clone
- 18. ADOP Concepts - Prepare • Optional Checks – Before running prepare • $ adop -validate – Run from primary (master) node, adop will perform the validation on all the available nodes – Run on a secondary (slave) node, adop will only run there – Checks for free space in the system » SYSTEM – min 25 GB free » APPS_TS_SEED – min 5 GB free – Run while a patching cycle is in progress • Validation will take place for the cutover phase • Prepare synchronizes patch filesystem from run filesystem
- 19. ADOP – 100% Online? • Is it ADOP 100% online? – No • How many tasks can be done online? – It depends • How much downtime to incur while patching? – It depends Photo by bruce mars on Unsplash
- 20. ADOP Concepts - Prepare • ADOP actions during prepare – Checks weather to perform a cleanup – Validates system configuration – Checks if DB is ready • APPS is edition-enabled (exits immediately) • Patch service (fix attempt) • Logon trigger (fix attempt) • DB data dictionary integrity (exist immediately) • ETCC – Checks all application nodes – "Online Patching In Progress" (ADZDPATCH) concurrent program – Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl – Checks the database for the existence of a patch edition – Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl Photo by Shane Aldendorff on Unsplash
- 21. ADOP Concepts - Prepare • Synchronization options – Just execute adop phase=prepare (DEFAULT) • Identify patches already applied to run APPL_TOP • Apply patches silently to patch APPL_TOP • skipsyncerror parameter – Create a new patch filesystem adop phase=fs_clone • Executed before prepare • Option force=yes – Delta? • adop phase=prepare sync_mode=delta • Delta synchronization of filesystems • Implemented using rsync on Unix and RoboCopy on Windows • Customizable
- 22. ADOP Concepts - Prepare • Synchronization Customizations – Patch synchronization - apply missing patches – Delta (file) synchronization - copy file changes – fs_clone synchronization - clone entire file system • Examples of patching actions that are not synchronized by default include: – Compiling user-defined JSPs – Copying some third-party libraries – Copying and compiling user-defined concurrent programs – Copying and generating user-defined forms
- 23. ADOP Concepts - Prepare • $APPL_TOP_NE/ad/custom/adop_sync.drv #Begin Customization ... #End Customization – The adop_sync.drv file is not currently reset to its template file at any point – You must check that the contents still meet your requirements • Multi-node – Execute on primary (Admin Server) node – ssh between primary and secondary nodes – If node is inaccessible it will be mark as abandonded
- 24. ADOP Concepts - Prepare • Configuration changes – If any configuration changes detected • Clone configuration from run to patch • Execute autoconfig on patch fs 09:46:46 00:00:00 ad.bin.adop EVENT Detecting configuration changes. 09:46:46 00:00:00 ad.bin.adop PROCEDURE [START] Looking for file system configuration changes 09:46:46 00:00:00 ad.bin.adop EVENT. File system configuration changes detected.
- 25. ADOP Concepts - Apply • ADOP applies specified patches to patch edition of DB and filesystem – adop phase=apply patchtop=<$PATCH_TOP> – merge=yes, listed patches will automatically be merged – Multiple patches can be applied with one command • adop phase=apply patches=2455636,2348464 – Apply patch and NLS • adop phase=apply patches=2454666,2454666_ESA:u245466.drv – Restart a failed patch • adop phase=apply patches=2455636,2348464 restart=yes • What about the failed worker? – Analytics. Generates in-depth reports • adop phase=apply analytics=yes
- 26. ADOP Concepts - Finalize • Used to perform any remaining processing needed to ensure the fastest possible cutover – Pre-compute DDL that needs to be run at cutover. – Compile all invalid objects. – Validate that the system is ready for cutover. – If finalize_mode=full, compute statistics for key data dictionary tables for improved performance. $ adop phase=finalize
- 27. ADOP Concepts - Cutover • Critical phase – application is down • Key actions – Shutdown concurrent managers. ADOP signals ICM to shutdown but wait for running requests • cm_wait=<maximum_minutes_to_wait> • Recommendations – Production - do not specify cm_wait – Non-Prod – specify cm_wait to limit the waiting time – Shutdown application tier services – Cutover database – Cutover file system – Terminate old database sessions – Start application tier services
- 28. ADOP Concepts - Cutover • $ adop phase=cutover • Deferring application tier restart – adop phase=cutover mtrestart=no • If failed and restarted cutover will continue at the failure point • JAR files – JAR files are stored in the $APPL_TOP/admin/<SID>/out directory and uploaded to the DB during cutover. • Autoconfig is executed
- 29. ADOP Concepts - Cleanup • If you fail to run cleanup phase explicitly it will be run during the next prepare • Drops (removes/cleans) obsolete – Crossedition triggers – Seed data – Editioned code objects (covered objects) – Indexes – Columns – Editions • 3 modes – cleanup_mode=quick – cleanup_mode=standard (default) – cleanup_mode=full
- 30. ADOP Concepts - Abort • Cancels/Terminates/Abandons the current patching cycle • It won’t apply any patches to the run FS (live) • It can only be used before the cutover phase • It drops the patch edition in the DB • Need to execute cleanup and fs_clone after abort $ adop phase=abort $ adop phase=abort,cleanup cleanup_mode=full Photo by Clem Onojeghuo on Unsplash
- 31. ADOP Concepts - FS_CLONE • fs_clone phase clones the run filesystem to the patch filesystem • Executed: – After ADOP Abort – At the end of the patch cycle – After implementing customizations on run filesystem $ adop phase=abort $ adop phase=fs_clone $ adop phase=prepare
- 32. ADOP Concepts - ACTUALIZE_ALL • Drop old database editions • Editions are accumulated after patch cycles • Recommended when 25 or more editions • This procedure will take large amount of time WARNING There are 25 existing editions. For optimum performance, there should be no more than 20. WARNING To remove unneeded editions, run the actualize_all phase and then perform a full cleanup. EVENT Validating system is ready to prepare. $ adop phase=prepare $ adop phase=actualize_all $ adop phase=finalize finalize_mode=full $ adop phase=cutover $ adop phase=cleanup cleanup_mode=full
- 33. Best Practices • Apply all recommended database and AD-TXK patches. How to Find E-Business Suite Recommended Patches (Doc ID 1400757.1) • Run ETCC and ensure the system is up to date • Disconnect any 3rd party applications prior the cutover phase • Use prepare sync_mode=delta $ adop phase=prepare sync_mode=delta • Be sure there’s no active patching cycle before running adpreclone • Turn the DB recycle bin off ######### # # recyclebin parameter # # The database recyclebin must be turned off to allow # the cleanup phase of the online patching cycle to be # performed without having to connect as SYS. # # This feature may still be used at other times. # #########
- 34. Best Practices • EBS Installation Configuration Management (ICM) Application DBA Online Patching (ADOP) Analyzer (Doc ID 2379656.1)
- 35. ETCC • E-Business Suite Technology Codelevel Checker (ETCC) • Helps identify overlay patches needed to be applied • Obtain ETCC Patch 17537119
- 36. Configuration Changes and Patching • Online configuration changes (recommended) – Execute prepare – Perform configuration changes to the patch filesystem • Weblogic configuration • HTTP (Apache) configuration • Filesystem changes – Execute cutover to promote those changes – Execute fs_clone • Offline configuration changes – Verify that no patching cycle is currently active – Shutdown application services – Perform configuration changes to run filesystem
- 37. Hot Patching? • NO! • 1987947.1, Granting Privileges On An Object May Cause Invalidations Photo by Gemma Evans on Unsplash
- 38. DB Patches • Most of the DB patches require the DB services to be down in the OH • Rolling patches if running a RAC DB • Recommend to install the DB patches after cutover $ adop phase=prepare sync_mode=delta $ adop phase=apply patches=4558512 $ adop phase=finalize $ adop cutover mtrestart=no SQL> shutdown immediate $ opatch apply SQL> startup $ adstrtal.sh
- 39. DR Patching • Most DB patches are DG Standby First Installable (README) • Install patch to your DR Standby DB first • Start ADOP Cycle • Install DB patch to primary after cutover • Install EBS patches to DR application • Run ADOP Cleanup
- 40. Issues • ADOP cycle slow – Verify if AD_ZD_DDL_HANDLER table is fragmented • Execute ADZDTABRDF.sql <owner> <table> • Gather table stats – SQL>exec dbms_stats.gather_table_stats('APPLSYS','AD_ZD_DDL_HANDLER’); – Old Data in ADOP repository tables • SQL>exec ad_zd_fixer.clear_adop_repo_tables; • Cutover Fails – Cutover_status=3 – 'Invalid Credentials' error – database patch edition has already been promoted to be the new run edition $ adop phase=cutover action=nodb
- 41. Blocking locks and Prepare Error on session xx: Creating database patch edition. Log: @ADZDSHOWLOG.sql Failed to execute SQL statement: declare result varchar2(10); begin ad_zd.PREPARE('Y’); … Error Message: ORA-20001: Error while calling ad_zd.prepare.ORA-04021: timeout occurred while waiting to lock object ORA-06512: at line 9 (DBD ERROR: OCIStmtExecute) [UNEXPECTED]Error creating patch edition. [UNEXPECTED]Error calling runCheckAndCreatePatchEdition subroutine. [ERROR] Prepare phase has failed.
- 42. Blocking locks and Apply • adop phase=apply error – Adop worker timeout – By looking at OEM huge red graph – This was a runaway Endeca session • Terminate the Endeca session • Restart the worker using adctrl (option 2) • Restart adop phase=apply • adop phase=apply completed correctly
- 43. Cutover Issues • Cutover hung – Verify if cutover is waiting for long requests to complete – See if those requests could be terminated • Cutover failing due to bug 27883586 – Issue with Mviews http://www.alfredokriegdba.com/2018/11/ebs-adop-phase-failing-due-to-ora-600.html • Cutover fails – Timeout while starting services on secondary node – $ adop phase=cutover mtrestart=no ADOP Cutover Fails, Timeout While Starting Services On Secondary Node (Doc ID 2411800.1)
- 44. Final Recommendations • Always, always! Keep your system up-to date with ETCC • Execute actualize_all regularly to cleanup old editions • Make use of adop phase=prepare sync_mode=delta • Be sure 3rd party applications are down before starting the prepare and apply if possible • Decided to shutdown concurrent managers before running cutover • Decided to execute cutover with mtrestart=no • Plan ahead and be prepared
- 45. Session ID: Remember to complete your evaluation for this session within the app! 10755 alfredo.krieg@viscosityna.com