![Oracle Real Application Testing NOCOUG Fall Conference Hanan Hit Director of Customer Support [email_address] Nov 13 2008 Taking the Risk out of IT Operations™ DBA InfoPower](https://image.slidesharecdn.com/RATNOCOUG111308-122757039735-phpapp03/85/NoCOUG-Presentation-on-Oracle-RAT-1-320.jpg)
NoCOUG Presentation on Oracle RAT
- 1. Oracle Real Application Testing NOCOUG Fall Conference Hanan Hit Director of Customer Support [email_address] Nov 13 2008 Taking the Risk out of IT Operations™ DBA InfoPower
- 2. Agenda Testing Intro The need for database testing Database Replay Workload Capture Workload Pre Processing Replay Workload Analysis & Reporting SQL Performance Analyzer Q&A
- 3. Why Test? Do you really want to go live like this?
- 4. Test What? Internally developed applications Packaged Applications even if you don’t posses the source code Remote Services even if you don’t know anything about how the services were written
- 5. Business/Scalability Graphs Value Old Value New
- 6. The Need For Database Testing Choose the right architecture – system changes are fact of life Hardware Software Deploy with confidence Get the system right at the first time, once and done Validate functionality/performance under real-world conditions Competitive Position Remove dependence of test cases on UI design Cut Test-Debug-Fix cycle time Apply a patch etc. Reduce testing and validation expense (both time and money ) Low success rate Causes of low success rate – Inability to test with real world production workloads
- 7. What is Real Application Testing? New database option available with EE only Includes two new features Database Replay (DB Replay) SQL Performance Analyzer (SPA) SQL Tuning Set (STS) functionality is shared between Real Application Testing option and EM Tuning Pack Advance functionality requires additional packs Diagnostic Pack for DB Replay Tuning Pack for SPA Price - Very Good Question ($11,500 per processor or $230 per named)
- 8. Value Proposition Value Reduces testing cost (Cheaper) Improves testing quality Business Benefit Faster technology adoption (Faster) Lower risk (Better) Solution for the Agile Business
- 9. Database Replay Recreate actual production database workload in test environment Identify, analyze and fix potential instabilities before making changes to production Capture Workload in Production Capture full production workload with real load & concurrency info Move the captured workload to test system Replay Workload in Test Make the desired changes in test system Replay workload with production load & concurrency Honor commit ordering Analyze & Report Errors Data divergence Performance divergence
- 10. Database Replay: Supported Changes Changes Unsupported Changes Supported Database Upgrades, Patches Schema, Parameters RAC nodes, Interconnect OS Platforms, OS Upgrades CPU, Memory Storage Etc. Middle Tier Storage Recording of External Client Requests Client Client … Client
- 12. Database Replay: Typical Configuration
- 13. Workload Capture All external client requests captured in binary files System background, internal activity excluded Minimal performance overhead for capture For RAC, shared and local file system supported Specify interesting time period for capture, e.g., peak workload, month-end processing, etc.
- 14. Workload Capture Steps Create a directory in order to save the captured files: mkdir–p /home/oracle/api_dir create directory API_DIR as ‘/home/oracle/api_dir’ Add Limits to the capture load In SQL*Plus (As system) begin dbms_workload_capture.ADD_FILTER(fname=> 'NO_SYS' , FATTRIBUTE=> 'USER', FVALUE=> 'SYS'); dbms_workload_capture.ADD_FILTER(fname=> 'NO_SYSTEM' , FATTRIBUTE=> 'USER' , FVALUE=> 'SYSTEM'); dbms_workload_capture.ADD_FILTER(fname=> 'NO_SYSMAN' , FATTRIBUTE=> 'USER' , FVALUE=> 'SYSMAN‘ dbms_workload_capture.ADD_FILTER(fname=> 'NO_SCOTT' , FATTRIBUTE=> 'USER' , FVALUE=> ‘SCOTT'); end; Shutdown and startup restrict the Prod DB (Not mandatory)
- 15. Workload Capture Steps Cont. Start the Capture begin dbms_workload_capture.start_capture(name=> 'CAPTURE-TSTDB01-20080306101900' , dir=>'API_DIR', AUTO_UNRESTRICT=> TRUE , DURATION=> NULL); end; Run load on the system Finish the Capture begin dbms_workload_capture.finish_capture(); end; Identify the ID of the capture select id , name , directory, start_time , end_time , status , duration_secs, dir_path from dba_workload_captures where rownum < 2 order by id desc ;
- 16. Workload Capture Steps Cont. Export the AWR Report – Make sure not to Flashback begin DBMS_WORKLOAD_CAPTURE.EXPORT_AWR(capture_id => <cap ID from previous step>); end ; Copy the Captures files to the test machine scp * oracle@192.168.1.61:/home/oracle/api_dir
- 17. Workload Capture Steps Cont.
- 18. Capture Options Workload can be filtered to customize what is captured Filter Types Inclusion Filters: specifies which sessions should be captured Exclusion Filters: specifies which sessions should NOT be captured Filter Attributes Workload capture can be filtered using any of the following session attributes User Program Module Action Service Session ID Workload capture can be run on-demand or scheduled to run at later time
- 19. Workload Pre-Processing Set up test system Logically similar data as production Use RMAN to physically restore production DB from backup Use Snapshot standby Use imp/exp, Data Pump, etc. Keep in mind Processing transforms captured data into replay files and generates necessary metadata Must be done on same version of database as replay system Recommended practice: process on test system Once processed, workload can be replayed many times For RAC, if using local file system, copy all capture files to single location for replay
- 20. Workload Pre-Processing Steps Process capture data on test database – Needs to be executed only once begin dbms_workload_replay.process_capture(capture_dir=>'API_DIR'); end;
- 21. Replay Workload File 1 File 2 File n … Metadata Replay Client is a special client program that consumes processed workload and sends requests to the replay system Special multi-threaded OCI client that replays the capture files Replay requests preserve timing, concurrency and dependencies seen on the capture system Replay Client consists of one or more clients. For workloads with high concurrency, it may be necessary to start multiple clients to drive workload. Once all replay clients are started, workload replay is then initiated by user Replay Files Replay Driver Replay Client Replay Client
- 22. Replay Options Synchronized Replay Workload is replayed in full synchronized mode Identical concurrency and timing as production workload Transaction commit order is honored Ensures minimal data divergence Unsynchronized Replay Workload can be replayed in unsynchronized mode Useful for load/stress testing High data divergence Three (3) parameters provided to control degree of synchronization Think time synchronization Commit order synchronization Connect (logon) time synchronization
- 23. Replay Options Unsynchronized Replay Parameters Think time synchronization Controls think time between database calls Auto (Default): Adjusts think time so as to maintain captured request rate Percentage 0% No think time, highest possible request rate <100% Higher request rate 100% Exact think time >100% Lower request rate Commit order synchronization Controls commit order between transactions In asynchronous mode, commit order not honored – transactions are committed as soon as commit call is issued Connect (logon) time synchronization Controls when sessions are created 0%: All sessioSn are connected immediately 100% (Default): Sessions connect at same time as in captured system
- 24. Replay Options Number of Replay Clients Configurable by user Client Calibration Advisor recommends number of replay clients needed for specific workload Replay clients are multithreaded clients that can drive multiple workload sessions each
- 25. Replay Workload Steps Initialize capture data on test database begin dbms_workload_replay.initialize_replay(replay_name=> LONGTEST ' , replay_dir=>'API_DIR'); end; Prepare capture data on test database begin dbms_workload_replay.prepare_replay; end; On the workload generation host(s), start the Workload Replay Clients. wrc system/<password>@TSTDB02 replaydir=/home/oracle/api_dir Start the reply process begin dbms_workload_replay.start_replay; end;
- 26. Replay Workload Steps Cont. Check Replay progress... Workload Replay Client: Release 11.1.0.6.0 - Production on Thu Mar 6 11:05:29 2008 Copyright (c) 1982, 2007, Oracle. All rights reserved. Wait for the replay to start (11:05:29) Replay started (11:06:10) Replay finished (11:59:00)
- 29. Analysis & Reporting Comprehensive reports are provided for analysis purposes Three (3) types of divergences are reported Data Divergence Number of rows returned by each call are compared and divergences reported Error Divergence For each call error divergence is reported New: Error encountered during replay not seen during capture Not Found: Error encountered during capture not seen during replay Mutated: Different error produced in replay than during capture Performance Divergence Capture and Replay Report: Provides high-level performance information ADDM Report: Provides in-depth performance analysis AWR, ASH Report: Facilitates comparative or skew analysis
- 30. Analysis & Reporting Steps Generate the AWR Report for the duration of the reply Generate the AWR comparison between two instances Must export the AWR from the source database Generate the ASH report from the test machine.
- 31. Workload Types Supported Supported All SQL (DML, DDL, PLSQL) with practically all types of binds Full LOB functionality (Cursor based and direct OCI) Local transactions Login/Logoffs Session switching Limited PL/SQL RPCs Limitations Direct path load, import/export OCI based object navigation (Abstract Data Types) and REF binds Streams, non-PL/SQL based AQ Distributed transactions, remote describe/commit operations Flashback Shared Server
- 32. Supported All SQL (DML, DDL, PLSQL) with practically all types of binds Full LOB functionality (Cursor based and direct OCI) Local transactions Login/Logoffs Session switching Limited PL/SQL RPCs Limitations Direct path load, import/export OCI based object navigation (Abstract Data Types) and REF binds Streams, non-PL/SQL based AQ Distributed transactions, remote describe/commit operations Flashback Shared Server
- 37. Required Privileges Any “Non-SYS” user with DBA Role SYSDBA role is not mandatory Does not have to be the user whose workload is captured “ Execute” privileges on DBMS_WORLOAD_CAPTURE/REPLAY procedures DBMS_WORKLOAD_CAPTURE START_CAPTURE, FINISH_CAPTURE, REPORT(), ADD_FILTER, DELETE_FILTER DBMS_WORKLOAD_REPLAY PROCESS_CAPTURE,INITIALIZE_REPLAY, PREPARE_REPLAY(), START_REPLAY(), CANCEL(), REPORT, ADD_FILTER, REMAP_CONNECTION Capture and Replay user can be different if they have appropriate privileges
- 38. … … Capture SQL Test impact of change on SQL query performance Capture SQL workload in production including statistics & bind variables Re-execute SQL queries in test environment Middle Tier Storage Oracle DB Re-execute SQL Queries Production Test Use SQL Tuning Advisor to tune regression SQL Performance Analyzer (SPA) Client Client … Client
- 39. Enables identification of SQL performance regressions before end users can be impacted SPA can help with any change that impacts SQL execution plan DB upgrades Optimizer statistics refresh New indexes, materialized views, partitions, etc. Regressed SQL can be fixed with SQL Tuning Advisor and SQL Plan Baselines Integrated with query optimizer Captures SQL workload with low overhead SPA Benefits
- 40. SQL Performance Analyzer Steps Capture SQL workload Move SQL workload to test system Execute SQL before making changes Execute SQL after making changes Compare & analyze performance
- 41. What’s Good Allows capture workload to be replayed on a test system to assess the impact of change Repetitive test scenarios at the database level Easy way to perform regression testing No need to use syntactic test values or homegrown scripts Was back-ported to 9i & 10g Requires less capital (Human & I&O) - lower operational costs The DBA role just got extended No Need for manual data entry
- 42. What’s Bad Multiple replays yields different DBID looks like system generated No way to parameterize the data Most of the issues are “non-database” issues Doesn’t have any business context – DBA tool Missing key processes for DSS systems (SQL Loader etc.) Supported only with EE Oracle Jobs can be very heavy Management can easily think that this is a good enough option to replace “legacy” scalability testing methods Think time between database calls assumes linear regression
- 43. Thank You Q&A “ If the facts don’t fit the theory, change the facts.” -Albert Einstein