Taming Oracle EBS R12.x Accrual Reconciliation Load
- 1. Session ID: Prepared by: Taming Oracle EBS R12.x Accrual Reconciliation Load Reduce Accrual Reconciliation Load Program Runtime Post R12.x Upgrade from EBS 11.5.10.2 10161 Joshua Johnson Sr. Oracle DBA Simmons Prepared Foods Inc. @jj_dba
- 2. Abstract Following our EBS R12.1.3 upgrade, we experienced inconsistent runtime and resource utilization with the Accrual Reconciliation Load program. During one month end close, a plant accountant could run accrual reconciliation for three years of data in under 10 hours. The next month, the same amount of data would take upwards of 30 hours to load. This behavior made it hard to plan for month end closure; successful accrual load being crucial to the finals for the month. Through multiple rounds of testing, we were able to complete the reconciliation process with nearly 15 years of data in just 1 hour and 45 minutes. 2
- 3. Taming Oracle EBS R12.x Accrual Reconciliation Load Who are we?
- 4. Presenters ■ Joshua Johnson ▪ Sr. Oracle Apps DBA — Simmons Prepared Foods, Inc. ■ Ned French ▪ Senior Manager — Centroid 4
- 5. 5 • Owned/Managed by Founding Partners Since 1997 • Centroid is one of Oracle’s Top 25 Strategic Partners/Platinum Partner Status • National technology firm focused on Consulting, Managed Services, Cloud Services and Resell of Oracle products that span the entire enterprise. • From Applications to Technology to Infrastructure; Centroid Data Center 100% Oracle Products: Apps, Tech, HW • AWARDED: 2014, 2015 Oracle Specialized Partner of the Year – North America; Engineered Systems Booth #815
- 6. Agenda ■ Brief background on our Oracle EBS upgrade from 11i to R12 ▪ Including hardware outline ■ OEM 12c Implementation ■ Server OS Level Modifications – Oracle Linux ▪ Huge Pages, Memory Limits ■ Oracle Database Modifications ▪ Indexes, PGA & SGA ■ Results – Test ■ Developing Our Plan of Attack ■ Maintenance Round 1 ■ Maintenance Round 2 ■ Results – Production ■ Current State ■ References 6
- 7. Our Oracle EBS Upgrade From 11i to R12 Upgrade from version 11.5.10 to 12.1.3
- 8. 11.5.10 Upgrade to R12.1.3 ▪ Database Server — OL 5.5 64-bit — 8 Older non-HyperThreading CPUs — 128 GB memory 8 ■ Upgrade completed using legacy Dell blade servers ▪ Application Server — OL 5.5 32-bit — 8 Older non- HyperThreading CPUs — 32 GB memory
- 9. 11.5.10 Upgrade to R12.1.3 Today marks 1010 days since Go Live! 9 Task Date Time Upgrade Start 03.JUL.2013 18:00 11i Pre-upgrade Functional Setup Complete 03.JUL.2013 23:59 12.1.1 Patching Complete 04.JUL.2013 22:00 12.1.3 Patching Complete 05.JUL.2013 04:00 Post Upgrade Patching Complete 05.JUL.2013 06:00 Business Owner Testing Complete 06.JUL.2013 17:00 All EBS User Accounts Reactivated 07.JUL.2013 17:00
- 10. OEM 12c Implementation Central repository for monitoring Oracle database instances
- 11. Painting a Rather Bleak Picture - DB Console 11 Create Accounting 06-JULY-2013
- 12. Painting a Rather Bleak Picture - DB Console 12 What happened here?
- 13. OEM 12c Implementation ■ Dbconsole ▪ Not meant to monitor a complex enterprise level production environment. ▪ Some servers are not currently configured to monitor and log errors. ▪ Unable to create the in- depth alerts necessary to proactively monitor an enterprise system. ■ OEM 12c ▪ Decreased time spent in reactionary processes ▪ Increased Oracle availability ▪ Improved system health and reporting ▪ Increased efficiency in generating reports for Oracle during an SR ▪ Increased ability to provide workload reports for our Application Developers and management staff. 13
- 14. Server OS Level Modifications – Oracle Linux • HugePages • Memory Limits
- 15. Server OS Level Modifications – Oracle Linux ■ Provide reserved, non-swappable memory space for our SGA ▪ 192 GB HugePages ▪ 60 GB PGA Target ▪ 100 GB SGA Target ■ Based on recommendations from the Oracle nmr.hugepages script. HugePages ■ Hard and soft memlock set to match 192 GB huge pages ▪ 201,326,592 kilobytes Memory Limits 15 $ grep -i huge /proc/meminfo HugePages_Total: 98304 HugePages_Free: 48276 HugePages_Rsvd: 1173 HugePages_Surp: 0 Hugepagesize: 2048 kB http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html /etc/security/limits.conf vm.nr_hugepages = 98304 /etc/sysctl.conf soft memlock 201326592 hard memlock 201326592
- 16. Server OS Level Modifications – Oracle Enterprise Linux 16http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html
- 17. Oracle Database Modifications • Indexes • PGA • SGA
- 18. Oracle Database Modifications Indexes Added to BOM, PO & XLA 18 ■ BOM.CST_WRITE_OFF_DETAILS_N2 ■ PO.PO_ACCRUAL_WRITE_OFFS_N5 ■ XLA.XLA_AE_LINES_N99 ■ XLA.XLA_AE_LINES_N5
- 19. Oracle Database Modifications Dropped Partitioned XLA Indexes 19 ■ XLA.XLA_AE_HEADERS_N1; ■ XLA.XLA_AE_HEADERS_N2; ■ XLA.XLA_AE_HEADERS_N3; ■ XLA.XLA_AE_HEADERS_N4; ■ XLA.XLA_AE_HEADERS_N5; ■ XLA.XLA_AE_HEADERS_N8; ■ XLA.XLA_AE_HEADERS_U1; ■ XLA.XLA_AE_LINES_N1; ■ XLA.XLA_AE_LINES_N4; ■ XLA.XLA_AE_LINES_U1; ■ XLA.XLA_DISTRIBUTION_LINKS_N1; ■ XLA.XLA_DISTRIBUTION_LINKS_N3; ■ XLA.XLA_DISTRIBUTION_LINKS_U1; ■ XLA.XLA_EVENTS_N1; ■ XLA.XLA_EVENTS_N2; ■ XLA.XLA_EVENTS_N3; ■ XLA.XLA_EVENTS_U1; ■ XLA.XLA_EVENTS_U2; ■ XLA.XLA_TRANSACTION_ENTITIES_N1; ■ XLA.XLA_TRANSACTION_ENTITIES_U1;
- 20. Oracle Database Modifications Recreated Partitioned XLA Indexes 20
- 21. Oracle Database Modifications Recreate script altered after encountering errors with the newsort feature in 11gR2 in testing 21
- 22. Oracle Database Modifications ■ SGA increased from 42 GB to 100 GB ▪ SGA and PGA configured to fit within the non-swappable hugepages. SGA Settings 22 pga_aggregate_target=60g sga_target=100g
- 23. Developing Our Plan of Attack • Maintenance Round 1 • Maintenance Round 2
- 24. Developing Our Plan of Attack ■ Not including OS and Database updates ▪ 17 hours required for maintenance in test. ■ We began testing options to separate these tasks into two smaller maintenance windows with the least amount of system downtime. How do we address the issue with the amount of time necessary to complete all of these maintenance items? 24
- 25. Maintenance Round 1 25 ■ Applied patch 15836954_R12.FND.B ■ Dropped partitioned indexes for XLA ■ Recreated partitioned indexes for XLA ■ Created new indexes for BOM, PO & XLA ■ Gather Schema Statistics for XLA, BOM, PO, AR and AP at 80% or better ■ Gather Table Statistics on XLA partitioned objects ■ Gather Table Stats on AP, PO, XLA
- 26. Maintenance Round 2 26 ■ Applied patches: ■ 10194427:R12.XLA.B ■ 20108234:R12.BOM.C ■ 14129023:R12.XLA.B ■ Gather Schema Statistics for XLA, BOM, PO, AR and AP at 80% or better ■ We restarted all EBS services after the patches applied successfully.
- 27. Results – Test • Test Database Hardware • Cisco Blade – 8 CPUs & 384 GB memory
- 28. Results – Test 28 Process Timing Apply Patch 15836954_R12.FND.B 45 minutes Drop XLA partitioned Indexes 2 minutes Recreate XLA partitioned Indexes 464 minutes Create New Indexes AP, PO & XLA 37 minutes Gather Schema Statistics for XLA, BOM, PO, AR, AP at 80% or better 120 minutes Gather Table Statistics on XLA partitioned objects Failed! Gather Table Stats on AP, PO, XLA Failed!
- 29. Results – Test Errors Encountered 29 Error Unable to recreate XLA_AE_LINES_N4 index manually or via the xdf script. Error from xdf run: ORA-17410 error occured during Index creation. Manually run: Error report - SQL Error: No more data to read from socket Solution We created a modified script to create the indexes manually with the newsort functionality disabled. "_newsort_enabled"=false; ■ Confirmed that there is sufficient space in the APPS_TS_TX_IDX tablespace prior to testing this solution.
- 30. Results – Production • Production Hardware • Dell Blade – 8 CPUs & 128 GB memory
- 31. Results – Production 31 Process Timing Apply patch 15836954_R12.FND.B 35 minutes Drop XLA partitioned Indexes 31 seconds Recreate XLA partitioned Indexes 271 minutes Create New Indexes AP, PO & XLA 33 minutes Gather Schema Statistics for XLA, BOM, PO, AR and AP at 80% or better. Gather Table Statistics on XLA partitioned objects. Gather Table Stats on AP, PO and XLA. These steps were combined into one SQL script. 733 minutes Applied patches 10194427:R12.XLA.B, 20108234:R12.BOM.C & 14129023:R12.XLA.B merged. 31 minutes Gather Schema Statistics for XLA, BOM, PO, AR and AP at 80% or better. 360 minutes
- 32. Production – Current State • New Production Hardware • Cisco Blade – 16 CPUs & 384 GB memory
- 33. Multiple Create Accounting Executions: 4-JAN-2016 33 ■ Multiple Create Accounting sessions for separate business units and their pre-processors were run in draft mode the morning of January 4, 2016. ■ At this time we were still enforcing the program incompatibilities. ■ They were also able to run Accrual Reconciliation Load without seeing an increase in system waits. In this system we are able to run Accrual Reconciliation Load for 15+ years of data in under 2 hours
- 34. 34 Multiple Create Accounting Executions: 4-JAN-2016
- 35. References
- 36. References ■ Oracle Database Administrator's Reference 11g Release 2 for Linux and UNIX-Based Operating Systems ■ E10839-23 ■ Oracle Database Performance Tuning Guide 11g Release 2 ■ E41573-03 ■ Oracle Database PL/SQL Packages and Types Reference 11g Release 2 ■ E40758-03 ■ Oracle Financials Implementation Guide Release 12.1 ■ E13425-05 ■ Oracle Linux Administrator's Solutions Guide for Release 6 ■ E37355-41 - January 2015 ■ Oracle Subledger Accounting Implementation Guide Release 12.1 ■ E13628-04 ■ Accrual Reconciliation Load Run Has Slow Performance (Doc ID 1490578.1) 36
- 37. Questions? 37
- 38. Thank You 38 You may complete the session evaluation either on paper or online via the mobile app Please complete the session evaluation We appreciate your feedback and insight Session ID: 10161