Enough Blame for System Performance Issues
1 like1,165 views
The document provides biographical information about Mahesh Vallampati including his career history working in IT roles for various companies, education background, white papers, blog, LinkedIn group, and contact information. It also discusses various performance issues that can occur with databases and applications and emphasizes the importance of properly identifying the root cause before blaming individuals or components. The case studies describe specific examples of performance problems encountered and the methods used to diagnose and resolve the issues.
Enough Blame for System Performance Issues
- 2. Mahesh Vallampati ◦ Career Director of Oracle Applications and Technologies at Coach America, the Bus Company. Sales and Consulting at Hotsos and SmartDog Director of IT Database Services at Ceva Logistics Sales and Consulting at Oracle for 9 years ◦ Education Courses in Business at Houston Community College Master’s in EE from Texas A&M University Career Focus ◦ Used to be DBA ◦ Now Techno-Functional (Fechnical)
- 3. White Papers ◦ http://www.slideshare.net/mvallamp Email: mvallamp@yahoo.com Blog: http://mvallamp.blogspot.com Linked in Group Leader: DBA Manager Oracle Alumni Admin for content: 5000 members
- 4. Performance Issues Blame Blame The Business Blame the Functional Team Blame the Developer Blame the DBA Conclusions
- 6. Performance Issues ◦ Described in vague terms The System is slow The disk is slow The network is slow The SQL sucks The server sucks The problem is in the Application server The database is slow The code is awful ◦ My favorite Slow as molasses
- 7. Issue Approach ◦ Sound sophisticated but not really effective (My opinion) The BCHR is low or high We need to run Stats-pack AWR is better No, ASH is better “My” method is better Have you run statistics? Have you rebuilt the indexes? Let us take a look at the top 10 SQL statements The average physical I/O is high
- 9. Let us Blame somebody Let us not leave anybody out Let us Blame ◦ The Business People ◦ The Functional Team ◦ The Developers ◦ The DBA’s
- 11. Manufacturing Company In the Midwest Migrated from Mainframe to Oracle Applications (11.5.10) Did Big Bang implementation of Manufacturing, Order Management and Financials System was barely usable Flew there the day they went live after receiving panic calls
- 12. Implementation vendor was there Another “expert” vendor was there ◦ BCHR and Block Size changes were being discussed Every theory possible was floating around Listened for an hour to the issues Asked a basic question ◦ Are you losing money and if so why?
- 13. $$$$ ◦ We are losing money ◦ We can’t take orders ◦ Order Management is not able to take orders It is taking 10 minutes to enter a 6 line order It used to take less than a minute in the mainframe Good place to start They can pay my bills if only they make money
- 14. Traced the Order Management process ◦ Entering a header ◦ Entering a line Line Trace ◦ Entering a line was taking 1 minute and a half Most of the time was spent on calculating the price of an item Enter “Advanced” Pricing
- 15. Query was 9 pages long Query executed 319,000 Logical I/O’s
- 16. Why are you using advanced pricing? We can make more money Let us try Simple Pricing Turned off Advanced Pricing 6 line Order with Simple Pricing took 1 minute Interesting!!!
- 17. Advanced Pricing Parameters for determining price ◦ Checks Overall credit Limit (AR) ◦ Checks outstanding balances (AR) ◦ Checks inventory levels (Inv) ◦ Checks overall order totals (Current Order) ◦ Checks payment history (AR)
- 18. Can we use Simple Pricing instead of Advanced Pricing? Enter the “Business” Discussion with COO ◦ Why are you using Advanced Pricing? We can make more money. ◦ How much more money can you make with Advanced Pricing instead of Simple Pricing? Don’t know
- 19. Potential Upside on a 1billion USD in revenue ◦ 4 million a year if we are lucky Investment required to make the system perform at acceptable levels ◦ Cost – 10 to 15 million dollars Standby Reporting Archive/Purge Hardware Investments Software Investments Investments in people, processes etc.
- 20. Probability 50% Discount Rate 5% Year1 Year2 Year3 Year4 Year5 Revenue Upside $4,000,000 $4,000,000 $4,000,000 $4,000,000 $4,000,000 NPV $17,317,907 NPV for 5 Years Probable Revenue $8,658,953 NPV times the Probability Investment Required $15,000,000 Capital Investment Required. Depreciation not included. Profit ($6,341,047) Revenue-Cost Incurred
- 21. Question to the COO ◦ Would you spend 15 million USD in today’s dollars for a potential upside of 4 million dollars ◦ Economic forecast is gloomy ◦ Interest and Borrowing costs are high Right Answer, No.
- 22. Changed from Advanced Pricing to Simple Pricing The system was fast in performing transactions System had over-capacity Even in month end, the system did not even flinch Performance team disbanded Rebuilding Indexes and tables into bigger block table-spaces got a yawn Solved in 2 days
- 23. There is not enough hardware in the world to support an irrational business requirement Ask why complex functionality is required when simple functionality will do Ask the “Business” to justify need for “Advanced Functionality” Solution ◦ Switch to Simple Pricing ◦ System Stabilized ◦ Project started on Monday Morning ◦ Project Ended Wednesday Afternoon ◦ Back Home Wednesday night
- 25. Manufacturing Company in Ohio Migrated to Oracle Applications Severe Performance Issues Order Management is slow Call center for customers is just horrific Each call taking 10-15 minutes when before each call during the mainframe was taking 2- 3 minutes Customer was using Configurator and BOM
- 26. Method-R Traced the Order Management Process which uses the Configurator Identified the SQL Statement that was taking a long time
- 27. SQL statement was performing close to 900,000 Logical I/O’s per execution SQL Statement was getting executed for every item in the BOM that makes up an order line The more complex the BOM, the more items in the BOM, the slower the process The SQL Statement was determining whether the item needed to be displayed in the invoice for the customer and in the packing slip
- 28. 1. SELECT 'x' 2. FROM BOM_BILL_OF_MATERIALS BOM, BOM_INVENTORY_COMPONENTS COMP, 3. MTL_SYSTEM_ITEMS_B MTL, MTL_SYSTEM_ITEMS_B MTL_BOM, MTL_PARAMETERS MP 4. WHERE BOM.COMMON_BILL_SEQUENCE_ID = COMP.BILL_SEQUENCE_ID 5. AND COMP.COMPONENT_ITEM_ID = MTL.INVENTORY_ITEM_ID 6. AND MTL.ORGANIZATION_ID = MP.ORGANIZATION_ID 7. AND (MTL_BOM.SEGMENT1 LIKE '%_ITEMS' 8. OR MTL_BOM.SEGMENT1 LIKE '%_ITEM' 9. OR MTL_BOM.SEGMENT1 LIKE '%_ITEMS2' 10. OR MTL.BOM_ITEM_TYPE = :B3 11. OR MTL.SEGMENT1 LIKE 'OC_%') 12. AND BOM.ASSEMBLY_ITEM_ID = MTL_BOM.INVENTORY_ITEM_ID 13. AND BOM.ORGANIZATION_ID = MP.ORGANIZATION_ID 14. AND BOM.ORGANIZATION_ID = MTL_BOM.ORGANIZATION_ID 15. AND MP.ORGANIZATION_CODE = :B2 16. AND NVL(COMP.EFFECTIVITY_DATE, SYSDATE) <= SYSDATE 17. AND NVL(COMP.DISABLE_DATE, SYSDATE) >= SYSDATE 18. AND COMP.COMPONENT_ITEM_ID = :B1
- 29. Should we tune the SQL? A couple of Predicates were using leading wildcards ◦ AND (MTL_BOM.SEGMENT1 LIKE '%_ITEMS' ◦ OR MTL_BOM.SEGMENT1 LIKE '%_ITEM' ◦ OR MTL_BOM.SEGMENT1 LIKE '%_ITEMS2' ◦ OR MTL.BOM_ITEM_TYPE = :B3 ◦ OR MTL.SEGMENT1 LIKE 'OC_%')
- 30. First instinct ◦ Blame the developer Met with the developer ◦ He said that was the functional spec Met with the functional person ◦ Asked them why they were looking for %ITEM %ITEMS %ITEMS2 OC_%
- 31. Functional Person explanation ◦ Oh, it is a convention we use ◦ If the item type ends in ITEM, ITEMS or ITEMS2 or starts with OC, then display it ◦ Otherwise, don’t display it ◦ They were proud of it too
- 32. Should we create a reverse string functional index and switch the predicates? Umm..No
- 33. Called a meeting ◦ Business Users ◦ Functional Team ◦ Developers ◦ DBA’s Took a dictionary to the meeting ◦ Asked them to find words that ended with M or S quickly ◦ Everybody was looking as if I was not sane I explained that was what they were asking the system
- 34. Created a flexfield with a Y or N at the inventory item level Updated the field to indicate the status Rewrote the SQL Statement to use the flag instead of the convention SQL now used 40 Logical I/O’s System speed was so fast that the productivity of the call center was up.
- 35. Try to answer the SQL statement yourself as if you did not have a system Would it make sense? If no, then the system is probably going to be bad at it too and will be slow Systems can process and automate irrational requests The cost and pain is going to be high System can do dumb but slow (Exadata anybody?)
- 37. Manufacturing Company in Illinois Customer had implemented Oracle Applications Shipping Application is slow SR with Oracle Support going nowhere Cannot determine if it is custom or standard
- 38. Method-R Traced the Shipping Process Each Line in the shipping screen was taking 20-25 seconds Identified the SQL Statement that was taking a long time Each execution was taking 650,000 Logical I/O’s.
- 40. SELECT fds.short_text FROM wsh_delivery_assignments wda, wsh_delivery_details wsh, fnd_attached_docs_form_vl fad, fnd_documents_short_text fds WHERE wsh.released_status = 'Y' AND wsh.delivery_detail_id = wda.delivery_detail_id AND wda.delivery_id = 124901 AND wsh.source_code = 'OE' AND fad.datatype_name = 'Short Text' AND fad.usage_type = 'O' AND fad.function_name = 'OEXOEORD' AND fad.entity_name = 'OE_ORDER_HEADERS' AND fds.media_id = fad.media_id AND fad.pk1_value = wsh.source_header_id AND ROWNUM = 1
- 41. 1 - filter(ROWNUM=1) 2 - filter("WDA"."DELIVERY_ID"=124901) 13 - access("DET"."DATA_OBJECT_CODE"='OE_ORDER_HEADERS' AND "DET"."LANGUAGE"=USERENV('LANG')) 15 - access("AF"."FUNCTION_NAME"='OEXOEORD') 16 - filter("DCU"."ENABLED_FLAG"='Y') 17 - access("DCU"."ATTACHMENT_FUNCTION_ID"="AF"."ATTACHMENT_FUNCTION_ID") 18 - filter("D"."USAGE_TYPE"='O' AND "DCU"."CATEGORY_ID"="D"."CATEGORY_ID") 19 - access("D"."DATATYPE_ID"="DAT"."DATATYPE_ID" AND "DAT"."LANGUAGE"=USERENV('LANG') AND "DAT"."USER_NAME"='Short Text') 20 - access("DCT"."CATEGORY_ID"="D"."CATEGORY_ID" AND "DCT"."LANGUAGE"=USERENV('LANG')) 22 - access("DT"."DOCUMENT_ID"="D"."DOCUMENT_ID" AND "DT"."LANGUAGE"=USERENV('LANG')) 24 - access("FDS"."MEDIA_ID"="DT"."MEDIA_ID") 25 - filter("AD"."ENTITY_NAME"='OE_ORDER_HEADERS') 26 - access("D"."DOCUMENT_ID"="AD"."DOCUMENT_ID") 27 - filter("WSH"."RELEASED_STATUS"='Y' AND "WSH"."SOURCE_CODE"='OE') 28 - access("WSH"."SOURCE_HEADER_ID"=TO_NUMBER("AD"."PK1_VALUE")) 29 - access("WSH"."DELIVERY_DETAIL_ID"="WDA"."DELIVERY_DETAIL_ID")
- 42. Oracle Financials has flexfields which have multiple purposes They have to be varchar2 It is a common practice to store key values in these columns Key values tend to be numbers Joins are made to flexfields from these keyfield If you use an implicit type conversion happening in the plan, this is usually the issue
- 43. SELECT fds.short_text FROM wsh_delivery_assignments wda, wsh_delivery_details wsh, fnd_attached_docs_form_vl fad, fnd_documents_short_text fds WHERE wsh.released_status = 'Y' AND wsh.delivery_detail_id = wda.delivery_detail_id AND wda.delivery_id = 1555559 AND wsh.source_code = 'OE' AND fad.datatype_name = 'Short Text' AND fad.usage_type = 'O' AND fad.function_name = 'OEXOEORD' AND fad.entity_name = 'OE_ORDER_HEADERS' AND fds.media_id = fad.media_id AND fad.pk1_value = to_char(wsh.source_header_id) AND ROWNUM = 1
- 44. Query now executes 700 Logical I/O’s down from 650,000 LIO’s Query now has sub-second response Productivity is now way up. Life is good
- 45. Known he was joining a varchar to a number field Recognize that it could be an issue Checked the number of Logical I/O’s it was doing Examined the explain plan Can we have a “lint” for SQL?
- 47. Customer was a food distributor in Illinois They were going live on Oracle Financials They had several customizations which they were rolling out Customizations were released as a series of scripts The scripts were rolled out at 2:30 PM Testing would start
- 48. Performance was horrible Find Customers, Find Orders and Find Shipment forms would take forever The next day the system performance was fine Everything was fast Everybody were stunned ◦ Client DBA team ◦ Oracle Consulting Team
- 49. Traced the processes using 10046 A lot of joins to custom temporary tables were observed for queries with poor performance A little bit more digging and almost all performance issues were related to the temporary tables So what is that with these temporary tables that performed well the day after they were deployed
- 50. After a few hours of searching, did a quick explain plan in SQL Developer Noticed the “Rule” in the explain plan Wait a minute – we should be using CBO. This is the 9i database Hmm..Let us look at the temporary tables Aaah..ah
- 51. Stats on temporary tables were not being collected as a part of the code deployment A nightly process that gathered stats on all objects that did not have statistics gathered the rest That explained why performance was poor the day the code was deployed and better the day after
- 52. This was interesting Whose job was it to have the stats gathering script as a part of the deployment Was it the DBA? Or the developer? It is a moot point when IT is on the line Is it the linebacker or the safety’s fault in a busted play? Who cares if you lose?
- 54. Use Method-R to get to the root cause Use other methods to confirm the problem ◦ Statspack ◦ Grid Control ◦ Homegrown scripts Avoid your first instinct
- 55. First Instinct ◦ Try to tune the SQL ◦ Try to solve the problem ◦ Wrong Approach Correct Approach ◦ Identify blame ◦ Assign Blame (Important) ◦ Ask the “makes sense” question ◦ Will it make us money? ◦ If not, don’t do it
- 56. Eliminate the problem Train the user not to create the problem Reschedule the problem Tune/Optimize the problem Relocate the problem