Thinking of Upgrading to Oracle SOA Suite 11g? Knowing The Right Steps Is Key (article)
0 likes474 views
The article discusses how an improperly modeled repository is often the cause of unsuccessful or delayed business intelligence (BI) implementations, as the repository plays a key role as the heart of any BI environment. It emphasizes that dimensional modeling principles should be followed when designing the repository to ensure requirements like reporting, performance, and user acceptance are met. Adhering to data modeling best practices when developing the physical and business models in the repository is also important for a successful BI implementation.
![Page 30 ■ 4th Qtr 2011
Tips, Notes and Information
from InSync 2011
By Tony Jambu
T
he Australian InSync 2011 conference was held
in Sydney in August. The format is similar to our
COLLABORATE conference in which the Australian OUG
and OAUG had a combined conference. Below are some of the
notes and tips from the conference and various sources.
Best InSync 2011 Presentation by Connor McDonald
Of all the presentations that I attended, the best presentation was from Connor
McDonald, Oracle ACE director and longtime OakTable member. The
presentation was titled “Managing Optimiser Statistics — A Better Way.”
It was the most entertaining and educational presentation I saw. The presentation
consisted of 360 slides. Yes, 360 slides! That’s not a typo. If anyone can present
360 slides in 45 minutes and still get the message across, Connor can.
If you are interested, the presentation can be read or downloaded from the
following link:
http://www.slideshare.net/InSync2011/databse-technology-2-connor-
mcdonald-managing-optimiser-statistics-a-better-waypdf
Here are just a few of his tips:
1) Statistics By Stealth
In Oracle 10g, the automatic system statistics gathering is carried out via the
GATHER_STATS_JOB job.
SQL select owner, job_name, enabled
2 from dba_scheduler_jobs
3 where owner = ‘SYS’;
OWNER JOB_NAME ENABLED
----------------------------------------------------
SYS PURGE_LOG TRUE
SYS FGR$AUTOPURGE_JOB TRUE
SYS GATHER_STATS_JOB FALSE
SYS AUTO_SPACE_ADVISOR_JOB TRUE
Running the same query in 11g produces
SQL select owner, job_name, enabled
2 from dba_scheduler_jobs
3 where owner = ‘SYS’;
OWNER JOB_NAME ENABLED
----------------------------------------------------
.
.
SYS AUTO_SPACE_ADVISOR_JOB FALSE
SYS GATHER_STATS_JOB FALSE
SYS PURGE_LOG TRUE
.
Here is where one would think that the system will not gather statistics on a
daily basis. The reality is that the default behavior is that the statistics will still
be gathered.
As Connor puts it, it is being gathered in “super stealth mode.”
SQL select client_name, status
2 from DBA_AUTOTASK_CLIENT;
CLIENT_NAME STATUS
------------------------------------ --------
auto optimizer stats collection ENABLED
auto space advisor ENABLED
sqltuning advisor ENABLED
These tasks are known as the “autotasks.” In 11g, Oracle introduced these sets
of autotasks. Further information can be obtained here:
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_
ID:1133388300346992024
http://download.oracle.com/docs/cd/B28359_01/server.111/b28318/
mgmt_db.htm#CNCPT1375
To turn off the automatic statistics gathering in 11g, you will need to call the
DBMS_AUTO_TASK_ADMIN package:
BEGIN
DBMS_AUTO_TASK_ADMIN.DISABLE(
client_name = ‘auto optimizer stats collection’,
operation = NULL,
window_name = NULL);
END;
/
2) Histograms
When gathering table statistics, it is possible to explicitly specify which
columns should have histograms statistics gathered. This would be tedious, as
it would require you to code the column name every time.
With Oracle 11g, it is now simplified with the DBMS_STATS.SET_SCHEMA_PREFS
and DBMS_STATS.SET__PREFS.
The SET_[SCHEMA|TABLE]_PREFS procedures allows you to set default values
that are used by the DBMS_STATS.GATHER_[SCHEMA|TABLE]_STATS.
As an example, you could set all tables to not gather any histogram statistics
with:
EXEC DBMS_STATS.SET_SCHEMA_PREFS(‘SCOTT’, ‘METHOD_OPT’, ‘for all columns size
1’);
SELECT Star](https://image.slidesharecdn.com/iougselectq411-160825002300/85/Thinking-of-Upgrading-to-Oracle-SOA-Suite-11g-Knowing-The-Right-Steps-Is-Key-article-32-320.jpg)
![4th Qtr 2011 ■ Page 31
3 where d.deptno= e.deptno
4 and e.jobid= 1
5 and e.salary 5000;
no rows selected
--------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |
--------------------------------------------------------------
| 0 | SELECT STATEMENT | | | |
| 1 | HASH JOIN | | 89 | 2136 |
| 2 | TABLE ACCESS FULL | DEPT | 1 | 10 |
|* 5 | TABLE ACCESS FULL | EMP | 4444 | 62216 |
--------------------------------------------------------------
Execution Plan
Note
-----
- cardinality feedback used for this statement
Notice that the execution plan above informs us that it had used the
cardinality feedback to improve its execution plan. Another way is to query
the USE_FEEDBACK_STATS column from V$SQL_SHARED_CURSOR view.
A nice enhancement to the Oracle optimizer would be for it to learn on the fly.
If the expected rows or expected CPU/time did not match the actuals, it would
stop and re-evaluate its plan with the new information and re-parse and
re-run the statement automatically. This feature is currently available in other
vendor’s optimizer.
IGNORE_ROW_ON_DUPKEY_INDEX Hint
Another well-known Oracle ACE Director and OakTable member presenting
at InSync 2011 was Richard Foote. His presentation was called “10 Things You
Might Not Know About Oracle Indexes.” There were lots of interesting facts
of which I was unaware. The presentation had many examples and demos to
help the audience comprehend the material being covered.
http://www.slideshare.net/InSync2011/database-technology-2-richard-
foote-10-things-you-probably-dont-know-about-oracles-indexespdf
One interesting piece of information was the topic on IGNORE_ROW_ON_
DUPKEY_INDEX hint. This hint was introduced in Oracle 11gR2.
http://download.oracle.com/docs/cd/E11882_01/server.112/e22487/
chapter1.htm#FEATURENO08842
http://download.oracle.com/docs/cd/E11882_01/server.112/e26088/
sql_elements006.htm#SQLRF30052
This is intended to assist with large INSERTs that may have duplicate records
as defined by the Unique Key. When inserting records without this hint
specified, any records that are duplicates will cause an ORA-00001 – unique
constraint error followed by rollback of all inserted rows.
However, with the IGNORE_ROW_ON_DUPKEY_INDEX hint specified, any duplicate
key rows will be ignored (with no warning) while the rest of the rows will be
accepted and inserted in to the table.
The syntax for using the hint is:
Followed by explicitly specifying which columns require histograms with:
EXEC DBMS_STATS.SET_TABLE_PREFS(‘SCOTT’, ‘EMP’, ‘METHOD_OPT’, ‘for column SAL
size 25’);
EXEC DBMS_STATS.SET_TABLE_PREFS(‘SCOTT’, ‘EMP’, ‘METHOD_OPT’, ‘for column DEMPNO
size 10’);
The next time you execute DBMS_STATS.GATHER_[SCHEMA|TABLE]_STATS for
Scott’s schema or EMP table, the SAL and DEPNO columns would gather
histogram statistics.
3) Oracle Cardinality Feedback
The Oracle optimizer is only as good as the input information or statistics that
it has. Sometimes it gets things wrong — especially when its estimated cardinality
(e-Rows) differs significantly from the actual rows (A-Rows) retrieved.
SELECT /*+ gather_plan_statistics */
COUNT(1) cnt, SUM(S.QUANTITY_SOLD) qty_sold, SUM(C.UNIT_COST) sum_cost
FROM sh.sales S,
sh.costs c
WHERE S.PROD_ID = C.PROD_ID AND s.time_id = :from_date
-------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time |
-------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:04.40 |
| 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:04.40 |
|* 2 | HASH JOIN | | 1 | 12M| 17M|00:00:03.45 |
| 3 | PARTITION RANGE ITERATOR | | 1 | 10699 | 15248 |00:00:00.01 |
| 4 | TABLE ACCESS BY LOCAL INDEX ROWID| SALES | 9 | 10699 | 15248 |00:00:00.01 |
| 5 | BITMAP CONVERSION TO ROWIDS | | 9 | | 15248 |00:00:00.01 |
|* 6 | BITMAP INDEX RANGE SCAN | SALES_TIME_BIX | 9 | | 17 |00:00:00.01 |
| 7 | PARTITION RANGE ALL | | 1 | 82112 | 82112 |00:00:00.06 |
| 8 | TABLE ACCESS FULL | COSTS | 28 | 82112 | 82112 |00:00:00.04 |
-------------------------------------------------------------------------------------------------------
But can Oracle learn from its mistake? With Oracle 11gR2, it can with its
Cardinality Feedback feature.
The following example is from Connor’s presentation. The first query was
execute and its plan is shown below.
SQL select e.empno, d.dname
2 from empe, dept d
3 where d.deptno= e.deptno
4 and e.jobid= 1
5 and e.salary 5000;
no rows selected
--------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |
--------------------------------------------------------------
| 0 | SELECT STATEMENT | | | |
| 1 | MERGE JOIN | | 4444 | 104K |
| 2 | TABLE ACCESS BY INDEX ROWID| DEPT | 100 | 1000 |
| 3 | INDEX FULL SCAN | DEPT_IX | 100 | |
|* 4 | SORT JOIN | | 4444 | 62216 |
|* 5 | TABLE ACCESS FULL | EMP | 4444 | 62216 |
--------------------------------------------------------------
With Cardinality Feedback working and learning that its actual rows did not
match its estimated cardinality, it would “remember” the actual values.
If you re-ran the same query with no changes, you would get:
SQL select e.empno, d.dname
2 from empe, dept d
continued on page 32](https://image.slidesharecdn.com/iougselectq411-160825002300/85/Thinking-of-Upgrading-to-Oracle-SOA-Suite-11g-Knowing-The-Right-Steps-Is-Key-article-33-320.jpg)
Thinking of Upgrading to Oracle SOA Suite 11g? Knowing The Right Steps Is Key (article)
- 1. Volume 18 | Number 4 Fourth Quarter 2011 For the Complete Technology & Database Professional w w w . i o u g . o r g BUSINESS INTELLIGENCE Understanding Oracle BI Components and Repository Modeling Basics by Abhinav Banerjee Finding Oracle Database Machine’s Rightful Place inYour IT Organization’s Arsenal by Jim Czuprynski Going Live on Oracle Exadata by Marc Fielding
- 2. My kind of oracle education & training • april 22-26, 2012 Mandalay bay, las vegas exclusives:With educational tracks devoted to: • bi/data Warehousing/epM • big data • database administration and development • Manageability • security, risk and compliance on hot topics like: • cloud computing • exadata • High availability • virtualization • dba101 full-day deep dives coMpliMentary for ioug registrants only! all other attendees pay the regular rate of $599. 8 hours of technical training, on topics like: • virtualization • Webcenter • business intelligence • rac • Manageability • big data • dba/developer • performance engineering exclusive access to Hands-on labs (a $350 value!) gain actual experience and turn theory into practice at 2-hour hands-on labs, focused on business intelligence topics like analytics, warehousing, and obiee. learn about exclusive ioug registration benefits & register: http://collaborate12.ioug.org collaborate 12 • registration opens noveMber 7, 2011 for under $2,000, collaborate 12 — the ioug (independent oracle users group) forum, offers access to over 1,000 hours of oracle-related education and training through Hands-on labs, boot camps, full-day deep dives, and customer case studies. exclusive ioug registration benefits
- 3. Executive Editor April Sims Associate Editor John Kanagaraj Asia-Pacific Technical Contributor Tony Jambu Managing Editor Theresa Wojtalewicz Associate Editor Alexa Schlosser Contributing Editors Ian Abramson Gary Gordhamer Arup Nanda Board Liaison Todd Sheetz How can I contribute to SELECT Journal? Write us a letter. Submit an article. Report on Oracle use in all corners of the globe. We prefer articles that conform to APA guide- lines. Send to select@ioug.org. Headquarters Independent Oracle Users Group 401 North Michigan Avenue Chicago, IL 60611-4267 USA Phone: +1.312.245.1579 Fax: +1.312.527.6785 E-mail: ioug@ioug.org Editorial Theresa Wojtalewicz Managing Editor IOUG Headquarters Phone: +1.312.673.5870 Fax: +1.312.245.1094 E-mail: twojtalewicz@ioug.org How do I get the next one? SELECT Journal is a benefit to members of the Independent Oracle Users Group. For more information, contact IOUG Headquarters at +1.312.245.1579 SELECT Journal Online For the latest updates and addendums or to download the articles in this and past issues of SELECT Journal, visit www.selectjournal.org. Copyright Independent Oracle Users Group 2011 unless otherwise indicated. All rights reserved. No part of this publication may be reprinted or reproduced without permission from the editor. The information is provided on an “as is” basis. The authors, contributors, editors, publishers, the IOUG and Oracle Corporation shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this publication or from the use of the programs or program segments that are included. This is not a publication of Oracle Corporation, nor was it produced in conjunction with Oracle Corporation. 4th Qtr 2011 ■ Page 1 C O N T E N T SVolume 18, No. 4, 4th Qtr. 2011 Features C O N T E N T S 5 Understanding Oracle BI Components and Repository Modeling Basics By Abhinav Banerjee Abhinav discusses how unsuccessful or delayed BI implementations are most often attributed to an improperly modeled repository not adhering to basic dimensional modeling principles. 12 Finding Oracle Database Machine’s Rightful Place in Your IT Organization’s Arsenal By Jim Czuprynski Jim explains how new capabilities in 11gR2 are likely to significantly improve the performance and throughput of database applications that can be leveraged for improved database application performance even without implementing an Exadata solution. 18 Going Live On Oracle Exadata By Marc Fielding Marc tells the story of a real-world Exadata Database Machine deployment integrating OBIEE analytics and third-party ETL tools in a geographically distributed, high-availability architecture. 22 Thinking of Upgrading to Oracle SOA Suite 11g? Knowing The Right Steps Is Key By Ahmed Aboulnaga Ahmed delves into how upgrading from Oracle SOA Suite 10g to 11g can be costly due to the dramatic change in the underlying architecture. This article takes you through a tried-and- tested upgrade strategy to help you avoid the pitfalls early adopters have faced. 28 Beating the Optimizer By Jonathan Lewis How do you access data efficiently if there’s no perfect index? Jonathan provides insight on how to creatively combine indexes in ways that the optimizer cannot yet manage and, by doing so, minimize the number of table blocks there are to access. 2 From the Editor 3 From the IOUG President 27 Users Group Calendar 30 SELECT Star 33 Advertisers’ Index 34 Quick Study Features Regular Features Reviewers for This Issue Dan Hotka Carol B. Baldan Kimberly Floss Sumit Sengupta Darryl Hickson Chandu Patel Aaron Diehl
- 4. Page 2 ■ 4th Qtr 2011 First, I would like to introduce a new contributing editor, long-standing IOUG volunteer Ian Abramson. His data warehouse expertise was a much-needed asset to round out our editorial board. There are also a few new features I would like to bring to readers’ attentions. We have added the list of reviewers to the Table of Contents to thank them for their dedication and hard work. SELECT Journal depends on reviewers to make sure each article’s technical details are correct and pertinent to readers. Another new feature, called Quick Study, allows SELECT Journal to give a nod to key volunteers at the IOUG. Readers are given a glimpse of the volunteers’ worlds through a few short questions. The PowerTips feature (see example on this page), SELECT Journal’s last enhancement, is a collection of small gems of knowledge throughout the magazine related to the overall theme of a particular issue. Q4 is focused on Business Intelligence, making Mark Rittman’s presentation at COLLABORATE 2011, “Oracle Business Intelligence: 11g Architecture and Internals,” a perfect choice for this role. 2012 Themes and Topics Announcement We are actively looking for new authors to write on the following topics: •• Q1 Theme: Tools for Change — Real Appplication Testing, SQL Performance Analyzer, Oracle Application Testing Suite, SQL Plan Management, Edition Redefinition •• Q2 Theme: Time for Change — Case Studies and Migrations: Non-RAC to RAC, Using Goldengate as an Upgrade, Non-ASM to Grid (RAC or Non- RAC), Exadata/Exalogic, 11gR2+ Upgrades •• Q3 Theme: Security — Oracle Database Firewall, Audit Vault, Oracle Label Security, Oracle Advanced Security, Hardening FMW11g, Oracle Entitlements Server •• Q4 Theme: Business Intelligence/Data Warehouse — Big Data, Performance, Advanced Features, Case Studies If you are interested in writing on any of these topics, email select@ioug.org. 2012 Print vs. Digital Debate This year, we have come up with the best compromise for digital versus paper copies of SELECT Journal. We are printing a hard copy of two editions for 2012: Q1 and Q3. The remaining two editions, Q2 and Q4, will be digital, with downloadable PDFs of all versions available online at http://www.ioug.org. Welcome to the Q4 2011 issue of SELECT Journal ! From the Editor Why are we doing this? It allows us to have paper copy of the magazine for the IOUG’s major events, COLLABORATE and Oracle OpenWorld. The digital version, on the other hand, allows for more features and longer articles. April Sims Executive Editor Another new feature, called Quick Study, allows SELECT Journal to give a nod to key volunteers at the IOUG. Readers are given a glimpse of the volunteers’ worlds ... Once again, we would like to thank all the authors and reviewers who contribute to SELECT Journal for their efforts in providing high-quality content. We always welcome your input and feedback, and we especially look forward to you sharing your expertise via this fine medium. Email us at select@ioug.org if you would like to submit an article or sign up to become a reviewer. April Sims Executive Editor, IOUG SELECT Journal BI Tip | Scale-out BI System If you use the “Scale-out BI System” option within the “Enterprise” install type to scale-out your OBIEE 11g system over additional servers, be aware that the embedded WebLogic license that you get with OBIEE 11g is for Standard Edition, not Enterprise Edition, which does not include support for WebLogic Clustering. Therefore, if you wish to use this horizontal scale-out feature, you’ll need to upgrade your WebLogic license, as a separate transaction, from Standard to Enterprise Edition, before you can legally use this feature. From Mark Rittman’s COLLABORATE 11 presentation “Oracle Business Intelligence 11g Architecture and Internals”
- 5. 4th Qtr 2011 ■ Page 3 For Exadata customers, we are working on developing educational content for online and in-person training offerings. We plan to launch a set of Exadata webinars providing a curriculum-based approach to Exadata education. The new IOUG Exadata SIG is active on LinkedIn and forming its web presence. New for COLLABORATE 12 Conference I always hear how critical it is to get out of the office and get some training. I’m happy to say that the IOUG continues to present, in partnership with Quest and OAUG, COLLABORATE 12 (April 22-26, 2012, in Las Vegas) Collectively, we bring to the community more than 1,000 peer-driven technical sessions that provide first-hand insight behind Oracle products. Full-week training can cost as much as $5,000, but for less than $2,000 and by registering through the IOUG, you can access: •• Free Full-Day Deep Dives (non-IOUG registrants cost is $595). Deep dives are held the Sunday before the conferences opens. See the listing of offerings for yourself, including topics on BI and Exadata. •• Exclusive Access to Hands-On Labs. These labs are included in the cost of your IOUG registration. Take a look at topics. •• IOUG Boot Camps. These are horizontal sessions that provide you the chance to learn about specialized topics for one or two full days during the conference. They’re led by great minds and included in the cost of your full conference registration. Back again this year for attendees who are fairly new to the job, we offer our popular DBA 101 Boot Camp. Registration for COLLABORATE 12 opens Nov. 7, 2011. As a user group, the IOUG exists for you and because of you. Whether you’ve recently joined or have been with us for years, I hope that we can be the source that you turn to, again and again, to solve problems, expand your knowledge, manage your career and, in short, make work life better. Sincerely, Andrew Flower IOUG President “IOUG provides great practical education and sharing of best practices on Oracle technology, especially Business Intelligence,” says Ari Kaplan, analytics manager for the Chicago Cubs and former president of IOUG. “I apply what I have learned over the years from IOUG to help mine through large amounts of data and find the most impactful and actionable information for our organization.” I hope you enjoy this issue! The IOUG is always very focused on a Business Intelligence curriculum, which is becoming increasingly important to all data management professionals. This issue is one great example of the caliber and quality of the educational offerings we provide. Thank you to our terrific editorial board for their support, leadership and vision on bringing this great content to the community! This Issue’s Feature: Business Intelligence Knowing that Business Intelligence is a significant area of interest to our members and the community, we’ve dedicated this issue of SELECT Journal to an examination of the process of translating data into a meaningful way of connecting the dots to drive business insight. In-depth as it may be, this journal is just the tip of the iceberg when it comes to the IOUG’s educational offerings. A quick tour of www.ioug.org reveals many other opportunities for you to become educated about the latest BI issues, including the IOUG Library, webinars, discussion forums, newsletters and the Tips & Best Practices Booklet — and, as a member, you can access all of this for free. Year-Round Education One of the best aspects of participating with the IOUG year round is access to a Special Interest Group (SIG). Whether you are grappling with the best way to implement a data warehouse that integrates information for the business and underpins your analytics; making recommendations on how to be more efficient delivering information; or looking to get the facts you need to improve the underlying technology, such as an investment in Exadata, there are others outside your company’s internal team that have similar interests and objectives. Participation in a Special Interest Group offers access to fellow members and Oracle product managers and engineers. IOUG BIWA (Business Intelligence and Data Warehousing) has thousands of members online and targeted content to bring together like-minded individuals. Andrew Flower IOUG President Dear Fellow IOUG Members… From the IOUG President
- 6. Database protection and compliance made simple IBM InfoSphere™ Guardium software is one of the most widely deployed solutions for continuously monitoring access to enterprise databases and simplifying compliance audits with automated and centralized controls for heterogeneous environments. • Help prevent data breaches, insider fraud and unauthorized changes to sensitive data. • Monitor privileged users such as database administrators, developers and outsourced personnel. • Virtually eliminate the overhead and complexity of native database management system audit logs. • Automate compliance reporting, vulnerability and configuration assessments, and data discovery. • Mask confidential data in test, training and development systems. For more information, visit ibm.com/guardium IBM, the IBM logo, ibm.com, and InfoSphere are trademarks of IBM, registered in many jurisdictions worldwide. A current list of IBM trademarks is available at ibm.com/legal/copytrade.shtml. Other company, product, or service names may be trademarks or service marks of others. © Copyright IBM Corporation 2011. All Rights Reserved.
- 7. 4th Qtr 2011 ■ Page 5 Understanding Oracle BI Components and Repository Modeling Basics By Abhinav Banerjee T he importance of Business Intelligence (BI) is rising by the day. BI systems, which help organizations make better and more informed decisions, are becoming crucial for success. There still are scenarios of huge BI investments going haywire; for example, multiple iterations of BI investments can exceed time and budget limits and implementations can fail user acceptance. One of the most common reasons for unsuccessful or delayed BI implementations is an improperly modeled repository (stores the metadata/business logic used by the BI server) not adhering to basic dimensional modeling principles. This article discusses this subject and describes the intricacies related to repository modeling and the associated concepts. Introduction In an Oracle Business Intelligence (OBI) implementation, the repository plays the most important role as the heart of any BI environment. The entire BI implementation can go wrong because of a repository that is not well- designed. RPD, or repository designing and modeling, is one of the most complex processes in an OBI implementation. RPD is based on knowledge of a few principles, which include dimensional modeling and data modeling. In any implementation, we need to ensure our data and dimensional models are well-designed. The data or dimensional model plays a significant role depending on the reporting requirements, which might be either operational or analytical. Once these models are in place, we need to ensure that the physical and the business models are properly designed and developed. It is highly recommended to have a well-designed dimensional model to ensure better performance even if you have a requirement for operational reporting; dimensional models are optimized for reporting, whereas the continued on page 6 traditional data-relational models are optimized for transactions. The complexity increases when requirements might include level-based measures, aggregates, multiple facts, multiple logical sources, conforming dimensions, slowly changing dimensions or very large data volumes. Dimensional Modeling (DM) DM refers to the methodology used to design data warehouses that need to support high performance for querying/reporting using the concept of facts and dimensions. Facts, or measures, refer to the measurable items or numeric values. These include sales quantity, sales amount, time taken, etc. Dimensions are the descriptors or the relative terms for the measures. Therefore, you have facts relative to the dimensions. Some of the most common dimensions include account, customer, product and date. Dimensional modeling includes the design of star or snowflake schema. Star Schema The star schema architecture constitutes a central fact table with multiple dimension tables surrounding it. It will have one to many relationships between the dimensions and the fact table. The dimensions typically have the relative descriptive attributes that describe business entities. In case of a star schema, no two dimensions will be joined directly; rather, all the joins between the dimensions will be through the central fact table. The facts and dimensions are joined through a foreign key relationship, with the dimension having the primary key and the fact having the foreign keys to join to the dimension. Snowflake Schema The snowflake schema architecture also has a central fact table with multiple dimension tables and one to many relationships between the dimension and the fact table, but it also will have one to many relationships between dimensions. The dimensions are further normalized into multiple related tables. In this case, multiple dimension tables will exist related to the main dimension table. Normally, we have one to many relationships between the dimensions. A primary key-foreign key relationship exists between the dimension and the fact tables as well as between dimensions. Oracle BI Architecture In order to understand the importance of the repository, we will need to have a look at the Oracle Business Intelligence Enterprise Edition (OBIEE) architecture. OBI repository directly corresponds with the Oracle BI server, which in turn talks to the database, presentation services and the security service. OBIEE is a state-of-the-art, next-generation BI platform that provides optimized intelligence to take advantage of the relational/multidimensional database technologies. It leverages the common industry techniques based on data warehousing and dimensional modeling.The OBIEE engine dynamically generates the required SQL based on the user’s inputs and designed model/definition in the repository to fetch data for the reports from the related databases. The various components of an OBI environment 11g, as shown in Fig. 1, include Java EE Server (WebLogic), Oracle BI Server, Oracle BI Presentation Services, Cluster Controller Services, Oracle BI Scheduler, Oracle Presentation Catalog, Oracle BI Repository, Security Service and BI Java Host. The various clients include Catalog Manager, BI Administration Tool, Scheduler Tool, Scheduler Job Manager, BI Answers and Interactive Dashboards. The next section takes a closer look at some of the major components within OBI 11g.
- 8. Page 6 ■ 4th Qtr 2011 presentation layer. Each layer appears in a separate pane when opened with the administration tool. Actions Services Actions services provides dedicated web services required by the action framework. The action framework enables users to invoke business process based on values of certain defined key indicators. It exists as action links in the presentation catalog. Security Service There is a paradigm shift in the security architecture in OBIEE 11g. It implements the common security architecture as the Fusion Middleware Stack, which leverages the Oracle platform security service (OPSS) and WebLogic authenticators. The various security controls that are available include: •• Identity Store — an embedded LDAP server in WebLogic to store users and groups •• Policy Store — a file to store the permission grants •• Credential Store — a file to store user and system credentials for interprocess communication Cluster Controller Servers There are two cluster controller servers in OBI 11g: a primary and secondary cluster controller. By default, they get installed in a clustered environment. Oracle BI Server Oracle BI server is the core behind the OBIEE platform. It receives analytical requests created by presentation services and efficiently accesses data required by the user using the defined metadata — RPD. BI server generates dynamic SQL to query data in the physical data sources and provides data to the presentation services based on the request received. It also works with the help of definitions in the configuration files and the metadata, which resides in repository, also referred to as RPD. Oracle BI Presentation Services Presentation services is implemented as an extension to a web server. It is deployed by default on OC4J, but Oracle supports additional web servers, such as WebLogic, WebSphere and IIS depending on the nature and scale of deployment. It is responsible for processing the views made available to the user and processes the data received from the BI server in an appropriate, user-friendly format to the requesting clients. There also is an associated Oracle BI Java Host service that is responsible for proper display of the charts and graphs. Presentation services uses a web catalog to store the saved content. Oracle BI Repository The repository has all the business logic and the design defined in it. It is the repository of the entire business logic/metadata. It can be configured through the Oracle BI administration tool. It helps build the business model and organize the metadata properly for presentation to users. The repository is comprised of three layers: physical, business model and mapping, and the Understanding Oracle BI Components and Repository Modeling Basics continued from page 5 Figure 1: OBIEE Enterprise Architecture
- 9. 4th Qtr 2011 ■ Page 7 the schema. Next, the connection pool needs to be defined in repository; details on how to connect to the database are stored in the OBIEE repository as shown in Fig. 3. Once complete, the physical layer will have the imported objects. It populates the connection pool with default values on import. Build Physical Model The next step is to build the physical model with the help of imported tables. It is here that we will define the objects and their relationships. To build the This provides a proper fallback environment in case of a single installation. The environment constitutes of a cluster controller and the cluster manager. Oracle BI Administration Tool The Oracle BI administration tool is the thick client used to configure the OBIEE repository. It allows viewing the repository in three separate layers: physical, business model and mapping, and presentation. The first step of the development process involves creating the physical layer. Oracle BI Development Cycle The development process begins with the creation of initial business requirements. You should have as many sessions with the business as possible to gather and confirm all the requirements. Try to look at the sample reports, spreadsheets, etc. Analyze the existing transactional system and the reporting system if any exist. Analyze the existing schema for the reporting system and the transaction system. Based on the requirements and the transaction schema, try to define the dimension model. There might be multiple iterations to the above steps. Build Oracle BI Model We can now look at how to build the Oracle BI model in the repository. Before we begin, the dimension model will need to be designed to meet the business requirements. In this section, I will explain the entire process of building the Oracle BI model. There are multiple parts to this process: import the objects if they don’t already exist in the physical layer; build the physical layer; build the logical-business model and mapping layer; build the presentation layer; and build the reports and dashboards based on the presentation-layer objects. Import Objects The first step involves creating a new RPD using the BI administration tool and saving it. Next, we must import the objects into this repository to start building the model as shown in Fig. 2. You will need to define the connection type and other credentials for connecting to the database. In order to import the tables, select the tables or just click on the schema name to bring them in continued on page 8 Figure 2: Sample Schema for Import Figure 3: Connection Pool Details Figure 4: Physical Model
- 10. Page 8 ■ 4th Qtr 2011 model, we need to create joins between the physical tables. At the physical layer we need to create foreign key joins — a sample is shown in Fig. 4. We should know the join criteria between the various tables. We need to maintain a 1:M relationship between the fact and the dimensions, which can be done by selecting the dimension first and then joining it to the fact. While creating the join, if the fact and the dimensions have the same keys, then by default they will appear in the Expression Builder. The expression shows the join criteria; Fig. 5 shows a sample. There is also a feature to have database hints that tell the database query optimizer to use the most efficient way to execute the statement, but we need to be very careful with this feature and use it after proper evaluation as it may have adverse impact in certain scenarios. This creates the join between the two selected tables. Similarly, we need to create joins between all the other dimensions and the fact. In the end, the physical model should look like Fig. 4. Next, we need to run a consistency check on the physical layer to ensure there are no errors related to syntax or best practices. If there are no consistency errors, we will see the consistency check manager screen with no error messages. Physical Layer Best Practices Here are some best practices that I have observed are important to follow to help your project be successful: •• You should have a separate set of connection pool for the initialization blocks and for the regular queries generated for the reports. This ensures a better utilization of the connection pools and ultimately results in performance improvements. •• Ensure that “Connection Pooling,”“Multithreaded Connections,” “Shared Logon” and the appropriate call interface is selected. •• You should not have any connection pools that cannot connect to the databases; this might lead to a BI server crash due to continuous polling of the connection. •• It is recommended to have a dedicated database connection — and preferably a system account — for OBI with access to all the required schemas. •• Always ensure that proper call interface is being used in the connection Understanding Oracle BI Components and Repository Modeling Basics continued from page 7 Figure 5: Foreign Key Join Figure 6: Logical Model pool definition. In the case of Oracle database, it’s better to use an OCI instead of an ODBC connection. •• Use the aliases of the tables instead of the main tables; this will avoid circular joins and caching-related issues. •• Follow a proper, consistent naming convention to identify the aliases, tables and columns. These may include W_XXX_F (for the fact tables), W_XXX_D (for the dimension tables), Dim_W_LOCATION_D_PLANT (for dimension alias tables) and Fact_W_REVN_F_ROLLUP (for fact alias tables). •• Always have cache persistence time set to ensure the data gets refreshed as required in case caching is enabled. Build BMM Model The next step is to build the business model and mapping (BMM) layer. In the development phase, this is the second step after creating the physical model. While designing the DM, though, it is the first step normally done before designing the physical model. Planning the business model is the first step in developing a usable data model for decision support. A successful model allows the query process to become a natural process by allowing analysts to structure queries in the same intuitive fashion as they would ask business questions. This model must be one that business analysts will inherently understand and one that will answer meaningful questions correctly. To begin, we need to give a name to the business model. Right-clicking in the BMM section of the window opens the following window, which allows the assignment of a name to the business model. The next step is to create the container for the business model. The easiest way to build the BMM layer is to either import in the tables from the physical layer or bring in the tables one by one as per requirements and then create the joins between them. In a complex environment, we normally do it one by one, as there might be multiple logical table sources, calculations and other customizations involved. Now we must look at the business model. To do that, we right-click on the HR model and select business model diagram. That will display the BMM diagram as shown in Fig. 6. The model is similar to the physical model. The major difference will exist in terms of the join criteria. We do not specify any joining columns in the logical layer; we only specify the cardinality and the type of join in this layer, as shown in Fig. 7.
- 11. 4th Qtr 2011 ■ Page 9 implementations, the number of consistency failures increases after an upgrade due to a lot of reasons. BMM Layer Best Practices To get the most from your OBI solution, each layer must be optimized. The following tips are some of my best practices that will help in the BM: •• Always use complex joins for joining the logical tables. Never use foreign key joins at the logical level as it might restrict the OBIEE server from using the most optimized path. •• Use inner joins wherever possible. Minimize the usage of outer joins, as they normally impact the performance. An easier solution for the problem of using outer joins is to build multiple logical table sources and, depending on the requirement, the appropriate logical table source is accessed. •• There should be a hierarchy defined for every logical dimension even if it only consists of a grand total and a detail level. •• If there is possibility of a hierarchy, then it’s always good to have a dimension hierarchy defined, as it helps to improve user experience. •• Ensure each level of the hierarchy has an appropriate number of elements and the level key. •• The lowest level of the hierarchy should be same as the lowest grain of the dimension table. The lowest level of a dimension hierarchy must match the primary key of its corresponding dimension logical tables. Always arrange dimensional sources in order of aggregation from lowest level to highest level. •• Give business-meaningful names in the BMM layer itself instead of making the changes in the presentation layer. •• Use aggregates if required and enable the aggregate rule for all measures. •• Aggregation should always be performed from a fact logical table and not from a dimension logical table. •• Columns that cannot be aggregated should be expressed in a dimension The administration tool considers a table to be a logical fact table if it is at the many end of all logical joins that connect it to other logical tables or if it’s not joined to any of the tables and the facts are displayed in yellow. As visible in Fig. 7, there are no expressions, so it picks up the base joins from the physical layer itself. Here in the logical layer we can configure the type of the join (inner, left outer, right outer, full outer) or the driving (fact or the dimension) and the cardinality. Cardinality defines how rows in one table are related to rows in the table to which it is joined. A one-to-many cardinality means that for every row in the first logical dimension table, there are possibly 0, 1 or many rows in the second logical table. Setting up the driving cardinality is an optional step; generally, it is set to none and left to the OBI server to process it. You should note that this option should be used with extreme caution; an improper configuration can lead to severe performance degradation. Driving tables are used in optimizing the manner in which the Oracle BI server processes cross-database joins when one table is very small and the other table is very large. Specifying driving tables leads to query optimization only when the number of rows being selected from the driving table is much smaller than the number of rows in the table to which it is being joined. Driving tables are not used for full outer joins. Also important to note here are the two entries in the features tab of database object that control and tune driving table performance: MAX_PARAMETERS_PER_DRIVE_JOIN and MAX_QUERIES_PER_ DRIVE_JOIN. The BMM layer allows you to create measures with custom calculations. You can build dimensional hierarchy by right-clicking on the dimension and selecting “Create Dimension.” Dimensional hierarchy is created for entities having two or more logical levels, a very common example being year, quarter, month and day. Once the customizations are finished, we need to do a consistency check before the business model can be made available for queries. The BMM object will have a red symbol until it passes the consistency check. If the connection is not working or objects have been deleted in the database, this utility will not report these errors. We can use the consistency check to test for errors, warnings and best practices violations. In certain Figure 7: Logical / Complex Join Figure 8: Custom Calculation continued on page 10
- 12. Page 10 ■ 4th Qtr 2011 logical table and not in a fact logical table. •• Nonaggregated columns can exist in a logical fact table if they are mapped to a source that is at the lowest level of aggregation for all other dimensions. •• The content/levels should be configured properly for all the sources to ensure that OBI generates optimized SQL queries. •• Create separate logical table sources for the dimension extensions. Build the Presentation Layer Once you are done with the physical and the BMM models, it is time to create the presentation layer. To begin, drag and drop the model from the BMM layer to the presentation layer. This approach can only be used when we have fairly simple models or are building a new model. Next, we will need to run another consistency check to ensure that the presentation layer and the repository are correct in terms of syntax and best practices. Before completing the development Understanding Oracle BI Components and Repository Modeling Basics continued from page 9 Figure 10: Repository Figure 9: Consistency Check
- 13. 4th Qtr 2011 ■ Page 11 •• Detailed presentation catalogs should have measures from a single fact table only as a general rule. The detailed dimensions (e.g., degenerate facts) are nonconforming with other fact tables. •• Never use special characters for naming convention in the presentation layer and dashboards. This completes the configuration of the repository. To use it, we will need to ensure that the BI server recognizes that this is the correct repository. That will require configuring the NQSConfig.ini and configuring the instanceconfig.xml to create a new presentation catalog and open your reporting solution to the end users for a robust and reliable experience. C ■ ■ ■ About the Author Abhinav Banerjee is a principal consultant working with KPI Partners. He has more than eight years of business intelligence and data integration experience with more than four years in OBIEE (custom and packaged analytics). He has worked with several global clients in various domains that include telecommunications, high tech, manufacturing, energy, education, and oil and gas. He is also a frequent speaker at various Oracle conferences such as COLLABORATE and Oracle OpenWorld. Abhinav specializes in OBIA as well as custom OBIEE implementations. He can be reached at abhinav1601@gmail.com. Figure 11: Usage of NQSConfig.ini cycle, we will need to take a few steps to clean the repository. We can remove all the columns not required for analysis, but we must keep in mind to not remove the keys from the logical dimensions, as the business model will not be valid. We should ensure that there are no extra objects in the repository; it helps with the maintenance and also keeps the repository light. Once done, the presentation layer will look as it does in Fig. 10. Presentation Layer Best Practices The final layer of the OBI solution is the presentation layer. The best practices that follow have improved the implementation of reporting: •• Ensure proper order of the objects so that it allows easy access to the required entities. •• Have business friendly/relevant names. •• Give a small description to serve as a tool tip for the users. •• Avoid designing the dashboard with large data sets. The requests should be quick and simple. •• Avoid using too many columns and use appropriate color combinations. •• Never combine tables and columns from mutually incompatible logical fact and dimension tables. •• Avoid naming catalog folders the same as presentation tables.
- 14. Page 12 ■ 4th Qtr 2011 Finding Oracle Database Machine’s Rightful Place in Your IT Organization’s Arsenal By Jim Czuprynski S ynopsis: The Exadata Database Machine offers intriguing opportunities to improve the performance of Oracle Database applications. The latest release of the Exadata X2-2 Database Machine incorporates several unique features that are bound tightly to Oracle Database 11gR2. This article delves into why these new capabilities are likely to significantly improve the performance and throughput of database applications. It also looks at how some of the features intrinsic to Oracle Database 11gR2 can be leveraged for improved database application performance even without implementing an Exadata solution. Exadata Database Machine: Basic Architecture and Concepts Oracle introduced the first version of the Exadata Database Machine at Oracle OpenWorld in October 2008. With the release of Database Machine X2 in 2010, however, it’s now touted as one of the fastest database platforms in the world based on its capability to process Oracle Database 11g Release 2 application workloads at blistering speeds (see Table 1). Oracle currently offers the Exadata Database Machine in three sizes: Quarter Rack, Half Rack and Full Rack. These machines combine flash memory solid-state storage, high-speed SAS hard disk drives (HDDs) and highly powered database servers. IDB, InfiniBand and ZDP Each Exadata Database Machine ships with Oracle Enterprise Linux (OEL) version 5 pre-installed as its OS and with Oracle 11g Release 2 RDBMS pre-installed as its RDBMS. The 11gR2 database kernel has been upgraded so that it can leverage several features unique to the Exadata Database Machine. The new iDB (INTELLIGENTDATABASE) communications protocol allows an 11gR2 database to communicate seamlessly and intelligently so that, when necessary, SQL query processing can be offloaded completely to the Exadata storage cells without having to retrieve all of the database blocks necessary to answer the query. Table 1. Exadata Database Machine: Rack ’Em and Stack ’ Em EXADATA DATABASE MACHINE Configuration X2-2 (“Quarter Rack”) X2-2 (“Half Rack”) X2-2 (“Full Rack”) DATABASE Servers 2 4 8 CPUs 24 48 96 Memory (GB) 192 384 768 Storage Servers 3 7 14 Number of CPUs 12 24 48 Number of Infiniband Switches 2 3 3 SAS Drive Capacity (TB): High-Performance (Raw / Useable) 21.0 / 9.25 50.0 / 22.5 100.0 / 45.0 High-Capacity (Raw / Useable) 72.0 / 31.5 168.0 / 75.0 336.0 / 150.0 Flash Drive Capacity (TB) 1.1 2.6 5.3 Theoretical Flash Cache IOPS 375,000 750,000 1,500,000 List Price (including support) $366,000 $671,000 $1,220,000 IDB also implements Zero-Loss Zero-Copy Datagram Protocol (ZDP). Constructed upon Reliable Datagram Sockets (RDS) version 3, this is open source networking software that’s also a zero-copy implementation of RDS that is more reliable than the User Datagram Protocol (UDP). Because an Exadata Database Machine uses 40 GBe Infiniband connections between its database servers and its storage servers, there is extremely little latency between when a database server communicates with its corresponding Exadata storage cells. Smart Flash Cache Smart Flash Cache is designed to overcome the limitations of individual hard disk devices (HDDs) whenever a database application’s random access response time requires a relatively large number of I/Os per second (IOPS) to satisfy customer service level agreements. Storage area networks (SANs) can overcome this limitation by placing dozens or even hundreds of HDDs in large arrays that have a combined random response time of more than 50,000 IOPS and then using large amounts of read/write cache to retain data for later reading if an identical section of a file is still available in cache. This also enables the SAN’s I/O subsystem to write the data back to disk at the best possible time to minimize write contention on individual physical disks. Exadata overcomes these limitations through its Smart Flash Cache features. Smart Flash Cache is implemented using PCIe-based single-level-cell (SLC) flash memory within each Exadata storage cell configured specifically for random access — especially reads — of identical database blocks. Oracle rates the 384GB of flash memory in an Exadata cell at 75,000 IOPS, and because multiple cells are linked together over the Infiniband network, they can perform huge numbers of random read operations in parallel. The largest Exadata Database Machine configuration — the X2-2 full rack — contains 14 Exadata storage cells, so it can theoretically achieve more than one million random-access read IOPS. Smart Flash Cache automatically will retain the most frequently accessed database blocks for both data and index segments, as well as the database’s control files and datafile headers. (Oracle DBAs who still want to use Smart Flash Cache in a similar fashion as they have been using the KEEP cache should note that Exadata does provide a special storage attribute for segments
- 15. 4th Qtr 2011 ■ Page 13 Not all SQL statements will be able to leverage storage indexes because only columns with a datatype of NUMBER, DATE or VARCHAR2 are supported. But there is a definite tendency in data warehousing and even OLTP processing for 90 percent of statements to be handled by 10 percent of the data, thus, the relatively small memory footprint for a storage index generally outweighs the alternative: unnecessary table scans of extremely large tables. And for partitioned tables, storage indexes offer yet another advantage. For example, consider a common situation where a table (INVOICES) has two columns that have an implicit relationship — for example, the date on which an invoice is issued (ISSUE_DATE) and the date on which it’s paid (PAID_DATE). It’s not unusual to partition the INVOICES table based on just one column (e.g. ISSUE_DATE); however, an Exadata storage cell can take advantage of a storage index to “prune” the result set further on the PAID_DATE column whenever it’s used in the predicate of a SQL statement. Yet another feature unique to Exadata storage cells, Smart Scan, allows the storage cell to return only the rows and/or columns necessary to satisfy a query. This is a significant alternative when an execution plan might normally return a huge number of database blocks because it needs to perform one or more table scans of large database tables. The storage cell instead scans the database blocks that need to be retrieved but then assembles just the rows and/ or columns into a result set that satisfy the request. Therefore, the processing of many SQL statements can be offloaded from the database server directly to one or more Exadata storage cells. Smart Scan processing incorporates two features: •• Predicate filtering lets a storage cell return only the rows necessary to satisfy a query rather than all the rows that would normally be returned when a table scan operation is needed. •• Like predicate filtering, column filtering (also known as column projection) returns only the columns necessary to answer a query’s request. It reduces the number of columns necessary to answer a query’s request, thus limiting the size of the result set returned to the database server. ASM Redundancy: Efficient Yet Robust Data Protection Since the initial release of Oracle’s Automatic Storage Management (ASM) file system in Oracle 10gR1, ASM’s feature set has improved dramatically. One of ASM’s most valuable features is the capability to provide two-way mirroring (NORMAL redundancy) or three-way mirroring (HIGH redundancy) for data stored within an ASM allocation unit (AU). Exadata storage cells leverage ASM’s NORMAL redundancy settings to provide essential data protection using JBOD (Just a Bunch Of Disks) HDDs instead of the relatively more expensive RAID that most enterprise storage systems provide. In addition, storage cells will leverage Smart Flash Cache whenever ASM needs to write the secondary AU that protects its counterpart primary AU. The secondary AU will not be cached in memory but instead can be written immediately to disk. ASM will instead read from the primary AU retained within Smart Flash Cache. When Will Exadata Most Likely Perform Miracles? It’s a rather foregone conclusion Exadata cannot help but reduce the execution times for reasonably well-tuned data warehousing queries, OLAP analysis and data mining operations. In fact, Oracle claims that some data warehousing queries will see reduction in query times by as much as two orders of magnitude (100x). For starters, Exadata’s Smart Flash Cache features almost guarantee that the database blocks needed to answer a query have already most likely been captured and retained within at least some of the storage cells’ flash cache memory. called CELL_FLASH_CACHE that will specifically retain any data, index or LOB segment within the Smart Flash Cache.) Smart Flash Cache is able to provide these features because it is intimately aware of not only which database blocks are stored within its confines, but also how database applications are actively using each database block. As a database block is retrieved from a storage cell and brought into the database buffer caches of a database server, Exadata retains the metadata of how the block is being utilized. Smart Flash Cache can then leverage this information to decide how the buffer should be retained within the cache and whether that buffer can satisfy other types of processing requests, including Recovery Manager (RMAN) backups, DataPump Exports, and especially Exadata’s Smart Scan and Smart Storage features. Hybrid Columnar Compression Unique to the Oracle Database Machine and Exadata, Hybrid Columnar Compression is a completely different way to store row-level data for a database table. Once enabled for a given table, Exadata first groups the rows into sets based on the similarity of values stored within the columns. These row sets are then tightly packed into logical storage structures called compression units. All the rows in a compression unit will contain similar values, and thus Exadata can compress the rows more quickly and also store them more efficiently. A compression unit contains row sets that encompass extremely similar column value ranges, so Exadata can also leverage this data homogeneity during SQL statement processing operations. Because this tends to add overhead to DML operations, HCC is best used on static or historical data. Storage Cells, Storage Indexes, Smart Scan and Smart Storage Exadata leverages Oracle’s Automatic Storage Management (ASM) for formatting and controlling all HDD and flash disk storage. Each individual storage cell is a combination of server, HDDs and even flash disks that can be constructed optionally. Utilizing a portion of the 384GB of flash memory, Exadata maintains a series of storage regions. These regions are automatically aligned on the same boundaries as ASM’s allocation units (AU). Each storage cell indexes these regions to retain metadata about the data distributions, storing this information within in-memory region indexes. Every region index can retain data distribution metadata for up to eight individual table columns. A storage index comprises one or more region indexes, so each storage cell is thus able to track in memory the value ranges for all data stored within all columns for all tables within that cell. Each storage cell automatically and transparently creates and uses storage indexes for all columns that appear to be well-clustered around similar column values. Therefore, the largest benefit typically will be obtained when a column’s data values are ordered within the table’s rows such that similar values are already closely clustered together, especially when an SQL statement will access rows using selection criteria predicates against those columns based on relatively simple equality (=), less than (<) or greater than (>) operators. Storage indexes also are destroyed whenever a storage cell is rebooted but will be rebuilt automatically after reboot as the storage cell sees fit, so there are no additional objects for a DBA to construct and maintain. Storage cells can thus use storage indexes to quickly skip much of the I/O processing that would normally be required with a traditional B-Tree or bitmap index. Without a storage index, it might be necessary to retrieve most or all rows of a table to determine if a query predicate can even be applied to the rows. And because a storage index provides intelligence on what’s already been retrieved from disk and already in cache within the storage cell, I/O may be completely unnecessary. continued on page 14
- 16. Page 14 ■ 4th Qtr 2011 Sorry, but bad SQL is still bad SQL. Poorly written SQL queries that unwittingly require millions of blocks to be accessed will probably run faster — or they may not. Using Exadata’s massively parallel architecture to improve the performance of over-HINTed, under-optimized or otherwise untuned SQL statements is like using a nuclear weapon to swat a fly: simply overkill. Points Of Serialization Must Still Be Located And Resolved. While it may indeed be possible to satisfy a larger number of user connections using Exadata’s Smart Flash Cache features, points of serialization may still exist within a poorly designed OLTP application. For example, an OLTP workload will still perform poorly in a RAC database environment whenever: •• Sequences are not used to obtain the next value for a “dumb number” primary key value; •• An insufficient number of sequence values are not cached on each RAC database instance to avoid possible contention during index row piece creation; and •• DSS or OLAP analyses are being executed simultaneously against the same database objects — particularly indexes — that are being actively modified via OLTP DML operations. Leveraging Non-Exadata Oracle 11gR2 Features To Simulate Exadata Performance Exadata’s capabilities to dramatically improve data warehousing and OLAP query speeds are certainly well-documented, but what if an IT organization’s current and future application workload profile really can’t benefit fully from those features? What if the application workloads are extremely diverse “hybridized” workloads that might benefit dramatically from only some of Oracle’s Smart Flash Cache features? Interestingly, it may be possible to utilize those features without incurring the full costs of an entire Exadata system. Oracle Database 11gR1 added several new enhancements that were most likely predecessors of the features enabling the tight complementarity of the Exadata Database Machine and 11gR2. Interestingly, these same features might make it possible for an astute DBA to reproduce significant performance improvements for their database application workloads. Oracle Advanced Compression In releases prior to Oracle Database 11gR1, table compression could only be applied to direct insertion of rows into an existing table via the COMPRESS FOR DIRECT LOAD storage directive. Oracle 11gR1 extended this support to include table compression for UPDATE and DELETE operations as well via the COMPRESS FOR ALL OPERATIONS directive. (In Oracle Database 11gR2, the DIRECT LOAD and ALL OPERATIONS directives have been renamed to BASIC and OLTP, respectively.) When activated for a table’s data segment, row pieces within the block will be compressed whenever the PCT_FREE threshold is reached. This compression continues until Oracle has compressed all row pieces within the block’s free space to their minimum size. The ultimate compressibility of row pieces within a block certainly depends upon the amount of CHAR and VARCHAR2 data and the number of “duplicate” values within the columns, but Oracle claims that in most cases compression ratios of 300 percent to 400 percent are not unlikely. Granted, this is still considerably less than the mega-compressibility that Exadata’s HYBRID COLUMNAR COMPRESSION feature offers, but in many cases it may significantly boost performance of OLTP and DSS applications because three to four times more rows can be read in a single IO operation since decompression is not needed for table scan operations. When a result set does need to be constructed, a storage cell is extremely likely to use Smart Scan features to construct it with extreme efficiency because only the rows and columns necessary to build it will be retrieved, even when a full table scan might be required to return the result set. In addition, Oracle claims that compression ratios of 10:1 are not uncommon with Hybrid Columnar Compression, so if a 200GB table did need to be scanned, as little as 20GB of disk I/O might be required. And because a single Exadata cell contains a relatively large number of CPUs, any query that can benefit from parallel processing will be able to take advantage of considerable CPU “horsepower” and will execute extremely quickly. In addition, if a query is executed with Exadata against a RAC database, then potentially all RAC database instances could “bring their guns to bear” to parallelize the query across those instances’ CPUs and memory. Therefore, a RAC database running in an Exadata environment should offer a significant opportunity to scale out parallelized query operations. Finally, should an Exadata cell ever need to actually read database blocks from its I/O subsystem, Exadata’s 40GBe Infiniband storage network means those blocks will be retrieved extremely quickly and with minimum overhead. When Should Exadata Perform Reasonably Well? As most OLTP applications apply changes to an extremely small number of database blocks when new rows are added or existing data is changed or removed, it’s usually much more difficult to improve the performance of OLTP workloads. Because it may be impossible to reduce significantly the number of blocks required to complete an OLTP transaction, extreme OLTP application workloads demand extremely low latency when communicating with the database server’s I/O subsystem. Each Exadata storage cell is benchmarked to provide 75,000 IOPS, and by extension this means a full-rack Exadata Database Machine can accommodate more than one million IOPS (14 storage cells x 75K IOPS = 1050K IOPS). This means that even a single full-rack Exadata Database Machine is uniquely positioned to provide the response times that extreme OLTP workloads typically demand. While Exadata’s Smart Flash Cache features do promote intelligent data placement on its underlying storage, an astute Oracle DBA often knows exactly which database segments — especially data and index segments — for an OLTP application workload would benefit from placement on the fastest I/O resources. Exadata’s Smart Data Placement feature gives an Oracle DBA the flexibility to place these known objects within the most appropriate storage for the application workload. Many Oracle shops have implemented RAC databases across several nodes to allow OLTP applications to scale up so that many thousands of concurrent user connections can be serviced simultaneously. For extreme OLTP application workloads, it’s crucial that the private interconnect networking layer between database servers is reserved exclusively for the intense demands of RAC Cache Fusion whenever buffers are exchanged between nodes in the RAC cluster. The good news here is that the Exadata Database Machine’s Infiniband 40GBe network is also used for the RAC private interconnect, which means it’s ideally suited for OLTP application workloads. When Might Exadata Yield Little or No Performance Improvements? There’s little doubt that Exadata offers an excellent platform for extreme DSS and OLTP application workloads. But an Oracle DBA should take a few into account when evaluating whether the considerable investment into the Exadata infrastructure will yield dramatic benefits for her application workloads: Finding Oracle Database Machine’s Rightful Place… continued from page 13
- 17. 4th Qtr 2011 ■ Page 15 Identifying Likely Candidate Tables For Compression Determining which tables or table partitions might benefit most from compression means it’s necessary to pinpoint which data segments are most heavily and frequently accessed during a typical application workload cycle: •• Automatic Workload Repository (AWR), first introduced in Oracle Database 10gR1, offers an excellent capability to pinpoint which tables and table partitions are most actively accessed through the Segment Statistics section of AWR reports and/or ASH reports. •• Another somewhat more manual alternative to using AWR performance metadata involves a simple report against the V$SEGMENT_STATISTICS dynamic view. •• Oracle Database 11gR2’s new DBMS_COMPRESSION package makes estimating the potential compression that any table or table partition might achieve a snap. Procedure GET_COMPRESSION_RATIO allows an Oracle DBA to calculate the potential average row length and block compression factor for OLTP compression. The results of this procedure’s execution against the SH.SALES data warehousing table for these compression factors is shown in the script available online. Smaller Is Better: Partitioning and Segment Space Allocation Improvements Oracle Database 11gR1 also introduced several new partitioning methods. An extension of the original RANGE partitioning method, the INTERVAL partitioning method now eliminates the need to attempt to predict the future size of each table partition; instead, the partition will be materialized only when a row is first added to the pertinent partition. Oracle 11gR2 expands upon this concept for nonpartitioned tables and indexes by providing the new SEGMENT CREATION DEFERRED storage directive, which creates the initial data segment for the corresponding table only when a row piece for a block within that segment is first created. Multicolumn Statistics As of Oracle 11gR1, it’s possible to gather extended statistics on multiple columns that encompass data that’s likely to be used simultaneously when rows are being selected. A few examples of these types of implicit data relationships include vehicle make and model (e.g., Ford Mustang) and geographical locations (e.g., Orlando, FL, USA). Whenever the Oracle 11g query optimizer detects that equality predicates are used against these multicolumn groups, it will use these extended statistics to build a better execution plan, thereby dramatically increasing the speed of data searches Poor Man’s Smart Scan: Caching Result Sets With +RESULT_CACHE While Exadata automatically creates and retains result sets for an application workload’s most commonly executed queries via Smart Flash Cache and Smart Scan, these features have already been available as of Oracle Database 11gR1. Yes, SQL Result Set Caching is not as automatic as Exadata’s implementation of Smart Scan, but any reasonably astute DBA can determine the most common queries that her application workloads are using though careful analysis of AWR reports. For example, she may identify that there’s a constant demand for up-to-date sales promotions data summarized within promotion categories, but she also knows that the underlying data for this query changes relatively infrequently and several user sessions could easily take advantage of a cached result set. The following code illustrates how simple it is to capture this result set into the database instance’s Shared Pool, thus making any part of these results available to any other query requesting these data: SELECT /*+RESULT_CACHE*/ promo_category_id ,promo_category ,SUM(promo_cost) total_cost ,ROUND(AVG(promo_cost),2) avg_cost FROM sh.promotions GROUP BY promo_category_id, promo_category ORDER BY promo_category_id, promo_category; Implementing Oracle Flash Cache in Oracle 11gR2 Many Oracle DBAs haven’t taken notice yet of one of the most revolutionary features in Oracle Database 11gR2: the ability to dramatically extend the size of their databases’ buffer caches by enabling Flash Cache. This feature set does require patching both Oracle 11gR2 Grid Infrastructure and 11gR2 Database homes to at least release 11.2.0.1.2, or by upgrading these homes to release 11.2.0.2.0. Once patched, it’s relatively simple to deploy Flash Cache against one of the many server-resident PCIe IO cards that have recently become available on the market. Table 2 lists two of the more popular high-capacity IO cards, including their manufacturer’s specifications and approximate list price. Table 2. Oracle Flash Cache Enablers Intra-Server IO Card Vendor FusionIO Virident List Price $14,000 $9,300 Memory Type (MLC, SLC) SLC SLC Card Size Half or Full Height Half or Full Height Card Capacity (GB) 400 350 Actual Capacity When Formatted (GB) 322.5 300 Speed Claims (Per Manufacturer) 119,000 IOPSi 300,000 IOPSii i 119K IOPS with 512 byte block size, 75/25 R/W ratio ii 300K IOPS with 4KB block size, 75/25 R/W ratio For the sake of simplicity, I’ll reference the FusionIO card to illustrate how an intra-server IO card can be formatted and configured for its eventual use by a single-instance database that’s also taking advantage of the new Grid Infrastructure features of 11gR2. I recently had the opportunity to experiment with a FusionIO card. I installed the card within a Hitachi BDS2000 blade server, configured it for use by installing the appropriate Linux device drivers, and a few minutes later I was able to format it as a physical IO device. I then used the native Linux fdisk command to create two new partitions on the device sized at approximately 28.6 and 271.8 GB, respectively: #> fdisk -l /dev/fioa Disk /dev/fioa: 322.5 GB, 322553184256 bytes 255 heads, 63 sectors/track, 39214 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/fioa1 1 3736 30009388+ 83 Linux /dev/fioa2 3737 39214 284977035 83 Linux At this point, the FusionIO card’s partitions can be utilized just as if it were any other physical storage device. The listing below shows the ASMCMD commands issued and the resulting output from Oracle 11gR2 Grid Infrastructure. continued on page 16
- 18. Page 16 ■ 4th Qtr 2011 I placed the 280GB partition of the FusionIO card into a single ASM disk group named +OFC: ASMCMD> mkdg ‘<dg name=”OFC”><dsk string=”/dev/fioa2” size=”280G”/></dg>’ Next, the database instance was restarted with a significantlyundersized database buffer cache of only 10MB. Note that Automatic Memory Management (AMM) and Automatic Shared Memory Management (ASMM) also were deactivated so that the instance could not dynamically allocate additional memory to the database buffer cache when it might run out of memory so that Oracle Flash Cache could be utilized fully during testing: SQL> ALTER SYSTEM SET DB_CACHE_SIZE=128M SCOPE=SPFILE; SQL> ALTER SYSTEM SET SGA_TARGET=0; SQL> ALTER SYSTEM SET MEMORY_TARGET=0; SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP; To activate Oracle Flash Cache as an extension of the instance’s buffer cache, I then modified just two parameters. DB_FLASH_CACHE_FILE determines the actual physical location of the Flash Cache file, and DB_FLASH_CACHE_SIZE restricts the ultimate size of the Flash Cache file itself. As illustrated below, I only had to specify an ASM disk group as the target for the file; once that’s done, the database will create a new physical file in the ASM disk group: SQL> ALTER SYSTEM SET DB_FLASH_CACHE_FILE=+OFC; SQL> ALTER SYSTEM SET DB_FLASH_CACHE_SIZE=280G; Conclusions It’s virtually impossible to question whether the Exadata Database Machine offers some absolutely incredible performance gains, especially for complex data warehousing queries, OLAP queries and data mining operations. Exadata also has the potential to dramatically improve the scale-up of OLTP application workloads — provided, of course, that the OLTP applications are truly scalable. But it would be equally unjust to promote Exadata as the ultimate panacea for improving the performance of all database application workloads. Some questions to help your team decide include: •• What is the I/O profile of the database workloads for the server(s) and storage subsystem(s) that Exadata is intended to replace? •• What are the minimum application workload performance improvement targets? •• What’s the cost/benefit ratio of implementing Exadata Database Machine, especially when the increased licensing costs are taken into account? •• What are the potential complexities of migrating existing Oracle databases to an Exadata environment, and is there a risk of any serious violations to application service-level agreements while the migration is completed? •• Finally, is your IT organization ready to accept at least in part the “one throat to choke” strategy that Exadata Database Machine implies? Or would simply deploying improved hardware (e.g., faster database servers, more server DRAM, SSDs and Flash Memory) enable the organization to improve application workload performance to exceed current service-level agreements? Finding Oracle Database Machine’s Rightful Place… continued from page 15 BI Tip | WebLogic Scripting Tool (WLST) If you want to script administration tasks usually carried out by Enterprise Manager, take a look at the WebLogic Scripting Tool (WLST) and the Oracle BI Systems Management API, which comes with features to change configuration settings, deploy repositories and perform most other OBIEE systems administration tasks, all from a Jython-based scripting environment. From Mark Rittman’s COLLABORATE 11 presentation “Oracle Business Intelligence 11g Architecture and Internals” The potential alternatives to a purely Exadata Database Machine solution presented in this article to solve common database workload performance issues are offered in Table 3 below. Even if an IT organization decides that the time for evaluating or implementing an Exadata solution is not on the future time horizon, these solutions offer insight into exactly how tightly coupled Oracle Database 11gR2 is with the storage solutions that only Exadata Database Machine offers: Table 3. Summary: Possible Alternatives to EXADATA Solutions Problem EXADATA Integrated Solutions Non-EXADATA Solutions Extending the database buffer cache’s capacity and performance Smart Flash Cache KEEP/RECYCLE caches Oracle Flash Cache Determining which objects to cache and where for most efficient usage Smart Scan Storage Indexes AWR Reports Segment Statistics Compressing rarely used data Hybrid Columnar Compression (Archival Compression) Oracle Advanced Compression (BASIC and OLTP) DBMS_COMPRESSION Compressing frequently used data Hybrid Columnar Compression (Warehouse Compression) Oracle Advanced Compression (BASIC and OLTP) Offloaded SQL Processing Smart Scan Storage Indexes SQL Result Set Caching Partitioning MultiColumn Statistics Recovery Manager (RMAN) Backups that support RTO/RPO requirements Block Change Tracking Incremental Level 1 Backups Massively Parallelized Multi-Piece Backup Sets (SECTION SIZE) C ■ ■ ■ About the Author Jim Czuprynski has accumulated more than 30 years of experience during his information technology career. He has filled diverse roles at several Fortune 1000 companies in those three decades before becoming an Oracle database administrator in 2001. He currently holds OCP certification for Oracle 9i, 10g and 11g. Jim teaches the core Oracle University database administration courses as well as the Exadata Database Machine administration course on behalf of Oracle and its education partners throughout the United States and Canada. He continues to write a steady stream of articles that focus on myriad facets of Oracle database administration at databasejournal.com.
- 19. © 2011 Embarcadero Technologies, Inc. All trademarks are the property of their respective owners. Introducing DB PowerStudio for Oracle It provides proven, highly-visual tools that save time and reduce errors by simplifying and automating many of the complex things you need to do to take care of your data, and your customers. DB PowerStudio™ for Oracle from Embarcadero gives you more Simplify Oracle Administration, Development, and Tuning Today. Get Free Trials and More at www.embarcadero.com Easier administration with DBArtisan® DB PowerStudio tools work across all versions of Oracle from a single interface. And with Embarcadero’s enterprise-friendly software licensing and delivery technologies, it’s easier to access, track, and manage tools licenses to minimize costs and maximize productivity. Faster performance with DB Optimizer™ Whether you already use Oracle Enterprise Manager (OEM) or some other third-party tool, you’ll find you can do many things faster with DB PowerStudio for Oracle. Simplified change management with DB Change Manager™ Faster development with Rapid SQL™
- 20. Page 18 ■ 4th Qtr 2011 Going Live On Oracle Exadata By Marc Fielding T his is the story of a real-world Exadata Database Machine deployment integrating OBIEE analytics and third-party ETL tools in a geographically distributed, high-availability architecture. Learn about our experiences with large-scale data migration, hybrid columnar compression and overcoming challenges with system performance. Find out how Exadata improved response times while reducing power usage, data center footprint and operational complexity. The Problem LinkShare provides marketing services for some of the world’s largest retailers, specializing in affiliate marketing, lead generation, and search1 . LinkShare’s proprietary Synergy Analytics platform gives advertisers and website owners real-time access to online visitor and sales data, helping them manage and optimize online marketing campaigns. Since the launch of Synergy Analytics, request volumes have grown by a factor of 10, consequently putting a strain on the previous database infrastructure. This strain manifested itself not only in slower response times, but also increasing difficulty in maintaining real-time data updates, increased database downtime and insufficient capacity to add large clients to the system. From the IT perspective, the legacy system was nearing its planned end-of-life replacement period. Additionally, monthly hard disk failures would impact performance system-wide as data was rebuilt onto hot spare drives. I/O volumes and storage capacity were nearing limits and power limitations in the datacenter facilities made it virtually impossible to add capacity to the existing system. Therefore, the previous system required a complete replacement. The Solution The end-of-life of the previous system gave an opportunity to explore a wide range of replacement alternatives. They included a newer version of the legacy database system, a data warehouse system based on Google’s MapReduce2 data-processing framework and Oracle’s Exadata database machine. Ultimately, Exadata was chosen as the replacement platform for a variety of factors, including the superior failover capabilities of Oracle RAC and simple, linear scaling that the Exadata architecture provides. It was also able to fit in a single rack what had previously required three racks, along with an 8x reduction in power usage. Exadata was able to deliver cost savings and improved coverage by allowing the same DBAs that manage the existing Oracle-based systems to manage Exadata as well. Once Exadata hardware arrived, initial installation and configuration was very fast, assured with a combination of teams from implementation partner Pythian; Oracle’s strategic customer program, Oracle Advanced Customer Services; and LinkShare’s own DBA team. In less than a week, hardware and software was installed and running. The Architecture User requests are handled through a global load balancing infrastructure, able to balance loads across datacenters and web servers. A cluster of web servers and application servers run Oracle Business Intelligence Enterprise Edition (OBIEE), a business intelligence tool allowing users to gain insight into online visitor and sale data from a familiar web browser interface. The OBIEE application servers are then connected to an Exadata database machine. 1 Affiliate Programs – LinkShare http://www.linkshare.com 2 MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean, Sanjay Ghemawat. http://labs.google.com/papers/mapreduce-osdi04.pdf Figure 1. Overall System Architecture Data flows from Oracle 11g-based OLTP systems, using a cluster of ETL servers running Informatica PowerCenter that extract and transform data for loading into an operational data store (ODS) schema located on the Exadata system. The ETL servers then take the ODS data, further transforming it into a dimensional model in a star schema. The star schema is designed for flexible and efficient querying as well as storage space efficiency. LinkShare’s analytics platform serves a worldwide client base and doesn’t have the off-hours maintenance windows common to many other analytics systems. The high availability requirements dictated an architecture (Fig. 1) that relies not on the internal redundancy built into the Exadata platform, but also to house two independent Exadata machines in geographically separated datacenter facilities. Rather than using a traditional Oracle Data Guard configuration, LinkShare opted to take advantage of the read-intensive nature of the analytics application to simply double-load data from source systems using the existing ETL platform. This configuration completely removes dependencies between sites and also permits both sites to service active users concurrently. In order to reduce migration risks and to permit an accelerated project timeline, application and data model changes were kept to a bare minimum.
- 21. 4th Qtr 2011 ■ Page 19 One of Exadata’s headline features is hybrid column compression, which combines columnar storage with traditional data compression algorithms like LZW to give higher compression ratios than traditional Oracle data compression. One decision when implementing columnar compression is choosing a compression level; the compression levels between QUERY LOW and ARCHIVE HIGH offer increasing tradeoffs between space savings and compression overhead.3 Using a sample table to compare compression levels (Fig. 2), we found the query high compression level to be at the point of diminishing returns for space savings, while still offering competitive compression overhead. In the initial implementation, a handful of large and infrequently accessed table partitions were compressed with hybrid columnar compression, with the remaining tables using OLTP compression. Based on the good results with columnar compression, however, we plan to compress additional tables with columnar compression to achieve further space savings. Performance Tuning Avoiding Indexes Improving performance was a major reason for migrating to Exadata and made up a large part of the effort in the implementation project. To make maximum use of Exadata’s offload functionality for the data-intensive business intelligence workload, it was initially configured with all indexes removed. (This approach would not be recommended for workloads involving online transaction processing, however.) The only the exceptions were primary key indexes required to avoid duplicate rows, and even these indexes were marked as INVISIBLE to avoid their use in query plans. Foreign key enforcement was done at the ETL level rather than inside the database, avoiding the need for additional foreign key indexes. By removing or hiding all indexes, Oracle’s optimizer is forced to use full scans. This may seem counterintuitive; full scans require queries to entire table partitions, as compared to an index scan, which reads only the rows matching query predicates. But by avoiding index scans, Exadata’s smart scan storage offloading capability can be brought to bear. Such offloaded operations run inside Exadata storage servers, which can use their directly attached disk storage to efficiently scan large volumes of data in parallel. These smart scans avoid one of the major points of contention with rotating storage in a database context: slow seek times inherent in single-block random I/O endemic in index scans and ROWID-based table lookups. Exadata storage servers have optimizations to reduce the amount of raw disk I/O. Storage indexes cache high and low values for each storage region, allowing I/O to be skipped entirely when there is no possibility of a match. The largest application code changes involved handling differences in date manipulation syntax between Oracle and the legacy system. The logical data model, including ODS environment and star schema, was retained. The legacy system had a fixed and inflexible data partitioning scheme as a by-product of its massively parallel architecture. It supported only two types of tables: nonpartitioned tables, and partitioned tables using a single numeric partition key, hashed across data nodes. The requirement to have equal-sized partitions to maintain performance required the creation of a numeric incrementing surrogate key as both primary key and partition key. The move to Oracle opened up a whole new set of partitioning possibilities that better fit data access patterns, all with little or no application code changes. More flexible partitioning allows improved query performance, especially when combined with full scans, as well as simplifying maintenance activities like the periodic rebuild and recompression of old data. The final partition layout ended up combining date range-based partitioning with hash-based subpartitioning on commonly queried columns. Data Migration Data migration was done in three separate ways, depending on the size of the underlying tables. Small tables (less than 500MB in size) were migrated using Oracle SQL Developer’s built-in migration tool. This tool’s GUI interface allowed ETL developers to define migration rules independently of the DBA team, freeing up DBA time for other tasks. Data transfer for these migrations was done through the developers’ own desktop computers and JDBC drivers — on a relatively slow network link — so these transfers were restricted to small objects. The table definitions and data were loaded into a staging schema, allowing them to be examined for correctness by QA and DBA teams before being moved in bulk to their permanent location. Larger objects were copied using existing Informatica PowerCenter infrastructure and the largest objects (more than 10GB) were dumped to text files on an NFS mount using the legacy system’s native query tools, and loaded into the Exadata database using SQL*Loader direct path loads. Simultaneous parallel loads on different partitions improved throughput. Initial SQL*Loader scripts were generated from Oracle SQL Developer’s migration tool but were edited to add the UNRECOVERABLE, PARALLEL and PARTITION keywords, enabling direct path parallel loads. The SQL*Loader method proved to be more than twice as fast as any other migration method, so many of the tables originally planned to be migrated by the ETL tool were done by SQL*Loader instead. (Although SQL*Loader was used here because of DBA team familiarity, external tables are another high-performance method of importing text data.) Another tool commonly used in cross-platform migrations is Oracle Transparent Gateways. Transparent gateways allow non-Oracle systems to be accessed through familiar database link interfaces as if they were Oracle systems. We ended up not pursuing this option to avoid any risk of impacting the former production environment, and to avoid additional license costs for a short migration period. One of the biggest challenges in migrating data in a 24x7 environment is not the actual data transfer; rather, it is maintaining data consistency between source and destination systems without incurring downtime. We addressed this issue by leveraging our existing ETL infrastructure: creating bidirectional mappings for each table and using the ETL system’s change-tracking capabilities to propagate data changes made in either source or destination system. This process allowed the ETL system to keep data in the Exadata systems up to date throughout the migration process. The process was retained post-migration, keeping data in the legacy system up to date. continued on page 20 3 Oracle Database Concepts11g Release 2 (11.2) Figure 2: Comparison of Compression Rates
- 22. Page 20 ■ 4th Qtr 2011 The Exadata smart flash cache uses flash-based storage to cache the most frequently used data, avoiding disk I/O if data is cached. The net result is that reading entire tables can end up being faster than traditional index access, especially when doing large data manipulations common in data warehouses like LinkShare’s. Benchmarking Performance Given the radical changes between Exadata and the legacy environment, performance benchmarks were essential to determine the ability of the Exadata platform to handle current and future workload. Given that the Exadata system had less than 25 percent of the raw disk spindles and therefore less I/O capacity compared to the legacy system, business management was concerned that Exadata performance would degrade sharply under load. To address these concerns, the implementation team set up a benchmark environment where the system’s behavior under load could be tested. While Oracle-to-Oracle migrations may use Real Application Testing (RAT) to gather workloads and replay them performance testing, RAT does not support non-Oracle platforms. Other replay tools involving Oracle trace file capture were likewise not possible. Eventually a benchmark was set up at the webserver level using the open- source JMeter4 tool to read existing webserver logs from the legacy production environment and reformat them into time-synchronized, simultaneous requests to a webserver and application stack connected to the Exadata system. This approach had a number of advantages, including completely avoiding impacts to the legacy environment and using testing infrastructure with which the infrastructure team was already familiar. A side benefit of using playback through a full application stack was that it allowed OBIEE and web layers to be tested for performance and errors. Careful examination of OBIEE error logs uncovered migration-related issues with report structure and query syntax that could be corrected. Load replay was also simplified by the read-intensive nature of the application, avoiding the need for flashback or other tools to exactly synchronize the database content with the original capture time. The benchmark was first run with a very small load — approximately 10 percent of the rate of production traffic. At this low rate of query volume, overall response time was about 20 percent faster than the legacy system. This was a disappointment when compared to the order of magnitude improvements expected, but it was still an improvement. The benchmark load was gradually increased to 100 percent of production volume. Response time slowed down dramatically to the point where the benchmark was not even able to complete successfully. Using database-level performance tools like Oracle’s AWR and SQL monitor, the large smart scans were immediately visible, representing the majority of response time. Another interesting wait event was visible: enq: KO – fast object checkpoint. These KO waits are a side effect of direct-path reads, including Exadata smart scans. Another session was making data changes — in this case updating a row value. But such updates are buffered and not direct-path, so they are initially made to the in-memory buffer cache only. But direct-path reads, which bypass the buffer cache and read directly from disk, wouldn’t see these changes. To make sure data is consistent, Oracle introduces the enq: KO – fast object checkpoint wait event, waiting for the updated blocks to be written to disk. The net effect is that disk reads would hang, sometime for long periods of time, until block checkpoints could complete. Enq: KO – fast object checkpoint waits can be avoided by doing direct-path data modifications. Such data changes apply only to initially empty blocks, and once the transaction is committed, the changed data is already made on disk. Unfortunately, direct-path data modifications can only be applied to bulk inserts using the /*+APPEND*/ hint or CREATE TABLE AS SELECT, not UPDATE or DELETE. Operating system level analysis on the storage servers using the Linux iostat tool showed that the physical disk drives were achieving high read throughput and running at 100 percent utilization, indicating that the hardware was functioning properly but struggling with the I/O demands placed on it. Solving the Problem To deal with the initial slow performance, we adopted a more traditional data warehousing feature of Oracle: bitmap indexes and star transformations.5 Bitmap indexes work very differently from Exadata storage offload, doing data processing at the database server level rather than offloading to Exadata storage servers. By doing index-based computations in advance of fact table access, they only retrieve matching rows from fact tables. Fact tables are generally the largest table in a star schema, thus, bitmap-based data access typically does much less disk I/O than smart scans, at the expense of CPU time, disk seek time, and reduced parallelism of operations. By moving to bitmap indexes, we also give up Exadata processing offload, storage indexes and even partition pruning, because partition join filters don’t currently work with bitmap indexes. With the star schema in place at LinkShare, however, bitmap indexes on the large fact tables allowed very efficient joins of criteria from dimension tables, along with caching benefits of the database buffer cache. The inherent space efficiencies of bitmap indexes allowed aggregatete index size to remain less than 30 percent of the size under the legacy system. After creating bitmap indexes on each key column on the fact tables, we ran the same log-replay benchmark as previously. The benchmark returned excellent results and maintained good response times even when run at load volumes of eight times that of the legacy system, without requiring any query changes. Query-Level Tuning Even with bitmap indexes in place, AWR reports from benchmark runs identified a handful of queries with unexpectedly high ratios of logical reads per execution. A closer look at query plans showed the optimizer dramatically underestimating row cardinality, and in turn choosing nested-loop joins when hash joins would have been an order of magnitude more efficient. Tuning options were somewhat limited because OBIEE’s SQL layer does not allow optimizer hints to be added easily. We instead looked at the SQL tuning advisor and SQL profiles that are part of Oracle Enterprise Manager’s tuning pack. In some cases, the SQL tuning advisor was able to correct the row cardinality estimates directly and resolve the query issues by creating SQL profiles with the OPT_ESTIMATE query hint.6 SQL profiles automatically insert optimizer hints whenever a given SQL statement is run, without requiring application code changes. OBIEE, like other business intelligence tools, generates SQL statements without bind variables, making it difficult to apply SQL profiles to OBIEE-generated SQL statements.A further complication came from lack of bind variables in OBIEE-generated SQL statements. Beginning in Oracle 11gR1, the FORCE_MATCH option to the DBMS_SQLTUNE.ACCEPT_SQL_PROFILE procedure7 comes to the rescue, matching any bind variable in a similar manner than the CURSOR_SHARING=FORCE initialization parameter. In many cases, however, the SQL tuning advisor simply recommended creating index combinations that make no sense for star transformations. In these cases, Going Live On Oracle Exadata continued from page 19 4 Apache JMeter http://jakarta.apache.org/jmeter/ 5 Oracle Database Data Warehousing Guide 11g Release 2 (11.2) 6 Oracle’s OPT_ESTIMATE hint: Usage Guide, Christo Kutrovsky. http://www.pythian.com/news/13469/ oracles-opt_estimate-hint-usage-guide/ 7 Oracle Database Performance Tuning Guide 11g Release 2 (11.2)
- 23. 4th Qtr 2011 ■ Page 21 we manually did much of the work the SQL tuning advisor would normally do by identifying which optimizer hints would be required to correct the incorrect assumptions behind the problematic execution plan. We then used the undocumented DBMS_SQLTUNE.IMPORT_SQL_PROFILE function8 to create SQL profiles that would add hints to SQL statements much the way the SQL tuning advisor would normally do automatically. Analyzing these SQL statements manually is a very time-consuming activity; fortunately, only a handful of statements required such intervention. Going Live LinkShare’s Exadata go-live plan was designed to reduce risk by slowly switching customers from the legacy system while preserving the ability to revert should significant problems be discovered. The ETL system’s simultaneous loads kept all systems up to date, allowing analytics users to run on either system. Application code was added to the initial login screen to direct users to either the legacy system or the new system based on business-driven criteria. Initially, internal users only were directed at Exadata, then 1 percent of external users, ramping up to 100 percent within two weeks. Go-live impacts on response time were immediately visible from monitoring graphs, as shown in Fig. 3. Not only did response times improve, but they also became much more consistent, avoiding the long outliers and query timeouts that would plague the legacy system. The second data center site went live in much the same manner, using the ETL system to keep data in sync between systems and slowly ramping up traffic to be balanced between locations. Operational Aspects Given that Exadata has a high-speed InfiniBand network fabric, it makes sense to use this same fabric for the I/O-intensive nature of database backups. LinkShare commissioned a dedicated backup server with an InfiniBand host channel adapter connected to one of the Exadata InfiniBand switches. RMAN backs up the ASM data inside the Exadata storage servers using NFS over IP over InfiniBand. Initial tests were constrained by the I/O capacity of local disk, so storage was moved to an EMC storage area network (SAN) already in the datacenter, using the media server simply as a NFS server for the SAN storage. Monitoring is based on Oracle Enterprise Manager Grid Control to monitor the entire Exadata infrastructure. Modules for each Exadata component, including database, cluster, Exadata storage servers, and InfiniBand hardware, give a comprehensive status view and alerting mechanism. This is combined with Foglight9 , a third-party tool already extensively used for performance trending within LinkShare, installed on the database servers. The monitoring is integrated with Pythian’s remote DBA service, providing both proactive monitoring and 24x7 incident response. Patching in Exadata involves several different layers: database software, Exadata storage servers, database-server operating system components like infiniBand drivers, and infrastructure like InfiniBand switches, ILOM lights-out management cards in servers, and even console switches and power distribution units. Having a second site allows us to apply the dwindling number of patches that aren’t rolling installable by routing all traffic to one site and installing the patch in the other. Looking Ahead With Exadata sites now in production, development focus is shifting to migrating the handful of supporting applications still running on the legacy system. Retirement of the legacy system has generated immediate savings in data center and vendor support costs, as well as freeing up effort in DBA, ETL and development teams to concentrate on a single platform. On the Exadata front, the roadmap focuses on making better use of newly available functionality in both the Exadata storage servers and the Oracle platform in general. In particular, we’re looking at making more use of Exadata’s columnar compression, incorporating external tables into ETL processes, and making use of materialized views to precompute commonly queried data. The Results The move to Exadata has produced quantifiable benefits for LinkShare. Datacenter footprint and power usage have dropped by factors of 4x and 8x, respectively. The DBA team has one less platform to manage. Response times have improved by factors of 8x or more, improving customer satisfaction. The ability to see more current data has helped users make better and timelier decisions. And, ultimately, improving customer retention and new customer acquisition. C ■ ■ ■ About the Author Marc Fielding is senior consultant with Pythian’s advanced technology group where he specializes in high availability, scalability and performance tuning. He has worked with Oracle database products throughout the past 10 years, from version 7.3 up to 11gR2. His experience across the entire enterprise application stack allows him to provide reliable, scalable, fast, creative and cost-effective solutions to Pythian’s diverse client base. He blogs regularly on the Pythian blog www.pythian.com/news, and is reachable via email at fielding@pythian.com, or on twitter @pythianfielding. Figure 3: Monitoring-server Response Times Before and After Exadata Go-Live 8 SQL Profiles, Christian Antognini, June 2006. http://antognini.ch/papers/SQLProfiles_20060622.pdf 9 Quest Software Foglight http://www.quest.com/foglight/
- 24. Page 22 ■ 4th Qtr 2011 Thinking of Upgrading to Oracle SOA Suite 11g? Knowing The Right Steps Is Key By Ahmed Aboulnaga U pgrading from Oracle SOA Suite 10g to 11g has proven to be very costly and challenging due to the dramatic change in the underlying architecture. This article presents a tried-and-tested upgrade strategy that will help you avoid the pitfalls early adopters have faced. Oracle SOA Suite 11g is used as the backbone for systems integration and as the foundation for integrating applications such as Fusion Applications. It is a comprehensive suite of products to help build, deploy and manage service-oriented architectures (SOA) and is comprised of products and technologies that include BPEL, Mediator and Web Services Manager. It also brings with it several great features, including support of the WebLogic Server, the introduction of the SCA model, improved conformance to standards and centralized access to artifacts and metadata. Unfortunately, error correction support ends on December 2013 for the latest release of Oracle SOA Suite 10g (specifically, 10.1.3.5). Thus, customers will have to choose between running on an unsupported release or upgrading to the latest version, currently SOA Suite 11g PS4 (11.1.1.5). Some customers erroneously believe that the new technology will resolve many of the pain points experienced in the older one, not realizing that a stabilization phase is still required. On the other hand, those who have invested (“suffered”) in stabilizing their current 10g environments may hold off on the upgrade because of the effort and risk involved. Let me be blunt. The upgrade process will be painful. Expect nearly all your code to require at least some change. A successful upgrade from SOA Suite 10g to 11g can only be achieved when both the development and infrastructure teams involved have a decent enough understanding of both versions, which is often not the case initially. A learning curve is inevitable, and typical training does not prepare you with the necessary upgrade knowledge. The effort involved in moving from Oracle SOA Suite 10g to 11g is both an upgrade and a migration. The upgrade is result of moving to a new version of the same product while the migration is the process of converting existing SOA Suite 10g code to allow it to deploy and execute on SOA Suite 11g. The entire process is sometimes unclear, laborious and introduces risk as a result of the underlying code being altered. The challenges of upgrading don’t end there. Several core concepts have fundamentally changed with the introduction of SOA Suite 11g, and there is not enough direction as to what to do. Understandably, Oracle documentation cannot cover every possible scenario. Based on recent implementation successes, IPN Web, a systems integrator based out of the Washington, D.C., area, has developed an approach to upgrading Oracle SOA Suite 10g to 11g that should suit most implementations while minimizing risk. This article summarizes that approach and highlights key areas not covered by Oracle documentation. Figure 1 is a flow diagram of the main steps involved in the upgrade process, as documented by Oracle. Unfortunately, this approach does little to actually help you in your project planning. Because Oracle Fusion Middleware 11g incorporates more than just SOA Suite, it is difficult for Oracle to produce a one-size-fits-all upgrade strategy. Secondly, despite the immense effort Oracle has put in the 11g documentation, and though it helps walk through various aspects of the code migration process, it is still lacking in many technical areas. The approach described in this article is more realistic and helps tackle this difficult upgrade project by providing clear, correctly sequenced steps. Furthermore, covered here are specific strategies not addressed by any Oracle documentation. The IPN Web SOA Suite 11g Upgrade Approach Although Oracle delivers great documentation on both Oracle SOA Suite and its upgrade process, it is not presented in a manner that is project-oriented and is missing certain areas required by all implementations. Our modified upgrade approach, based on Figure 2, is as follows: Figure 1. The Oracle Fusion Middleware Upgrade Process
- 25. 4th Qtr 2011 ■ Page 23 5. Code Promotion Follow your release management process to promote and test the code to higher-up environments, after which the upgrade/migration is officially over. Although this article does not address a production cutover strategy, it definitely should be considered during your planning phases. But by following these five steps, the challenges faced when upgrading is minimized. The rest of this article provides additional details and offers insight into areas not addressed by the Oracle documentation. 1. Understand the Upgrade Process The approach proposed focuses specifically on SOA Suite (and not the entire Fusion Middleware stack), does not mandate strong experience in SOA Suite 11g and provides realistic steps to follow. Review Existing Oracle Documentation Prior to engaging in the upgrade project, it is important to obtain an overall understanding of the upgrade process by reviewing existing Oracle documentation, which helps in the planning stages and further points you to other appropriate technical documentation. The documents listed below introduce the reader to the “Supported Upgrade Starting Points for Oracle Fusion Middleware 11g.” They also assist in developing an upgrade strategy, highlight key installation and upgrade tools, and document special considerations for an Oracle SOA Suite upgrade. These documents include: •• Oracle Fusion Middleware Upgrade Planning Guide 11g Release 1 (11.1.1) •• Oracle Fusion Middleware Upgrade Guide for Oracle SOA Suite, WebCenter, and ADF 11g Release 1 (11.1.1) •• Oracle Fusion Middleware Developer’s Guide for Oracle SOA Suite 11g Release 1 (11.1.1) 2. Prepare the Development Environment Installing and configuring the development environment should be completed early on, giving the developers the ability to start working on their pieces as soon as possible. Infrastructure Installation Oracle SOA Suite 11g is a new installation (not an in-place upgrade of the existing SOA Suite 10g); thus, the preferred approach is to maintain both environments in parallel for the duration of your project. This simplifies development, testing and the cutover process. Installing Oracle SOA Suite 11g in a single-node architecture is rather straightforward and typically involves: •• installing Oracle Database 11g; •• installing Oracle WebLogic Server 11g; •• creating SOA database schemas using the Repository Creation Utility (RCU) 11g; and •• installing Oracle SOA Suite 11g and creating the SOA domain. Though SOA Suite 11g is comparable in performance to 10g, memory requirements are typically much higher, particularly when the number of composites and shared schemas grow. Installing the product set on a 64-bit operating system with a 64-bit JVM is highly recommended. Though this may result in a slight performance hit, the scalability (in terms of throughput, message size, etc.) outweighs this slight disadvantage. 1. Understand the Upgrade Process Start by reviewing what may seem as overwhelming Oracle documentation to familiarize yourself with both the high-level process and individual steps. “The Oracle Fusion Middleware Upgrade Planning Guide 11g,” discussed briefly later in this article, familiarizes the reader with the upgrade strategy and tools. 2. Prepare the Development Environment Though obvious in nature, quickly standing up a single-node development environment allows for the developers to immediately begin the code migration process. Typically, the deployment process is developed and firmed up later on in the project. Here, we recommend finalizing it early on. The reason for this is that SOA Suite 11g introduces the MDS, or metadata services, which is a database-based repository for storing artifacts including schemas, WSDLs, DVMs and fault policies. Without these artifacts in the MDS, developers are unable to reference them, and the only way to import them into the MDS is through ant. Thus, understanding and deploying these artifacts at this stage is deemed critical. 3. Migrate the Code Migrating the code involves pre-migration, migration and post-migration tasks. We address design considerations that Oracle documentation does not cover due to the obsolescence of a number of features and functionality such as certain ESB header functions, BPEL domains and ESB systems and service groups. 4. Install Remaining Environments Typically your production environment will be clustered to support high availability and failover, as documented in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite 11g. At this point, the SOA application administrators have the time to familiarize themselves with the Oracle Enterprise Deployment Guide in parallel of the code migration to install, set up and configure those environments. Figure 2. The IPN Web Environment Upgrade and Code Migration Process for Oracle SOA Suite continued on page 24
- 26. Page 24 ■ 4th Qtr 2011 Systems and service groups are used in ESB 10g to group ESB services. Partitions, unlike ESB Systems and Service Groups, do not have listeners. However, from a code-grouping aspect, they can be treated similarly to ESB Systems and Service Groups. Services Groups in ESB 10g can be cascading (i.e., a service group can host another service group), but that is not the case in 11g. Thus, cascading hierarchies will need to be flattened. For example, if you have a system called “HR” with two service groups —“Europe” and “United States” — it will have to be flattened as shown in Figure 4. Partition names cannot have spaces. Figure 4. ESB 10g Systems and Service Groups (left) and Oracle Fusion Middleware 11g partitions (right) Migrate BPEL Domain Properties There is technically a single domain in Oracle SOA Suite 11g (versus potentially multiple domains in SOA Suite 10g), so properties will affect the entire server. Table 1 shows properties that have changed between 10g and 11g. Table 1. BPEL properties that have changed from 10g to 11g 10g Property 11g Property dspInvokeAllocFactor No longer exists dspMaxThreads No longer exists processCheckSecs No longer exists txDatasourceJndi No longer exists uddiLocation oracle.soa.uddi.registry.inquiryUrl uddiPassword oracle.soa.uddi.registry.password uddiUsername oracle.soa.uddi.registry.username Develop the Deployment Process Although not described in any Oracle documentation, we strongly recommend that all ant scripts be prepared at this time. This is not only needed when importing artifacts to the MDS (as required in the next step), but it will simplify and speed up your future deployments. SOA Suite 11g provides great ant script to ease the deployment process. The most important targets needed for your upgrade project are highlighted in Table 2. Table 2. Key ant targets needed for the upgrade Description Ant Script Ant Target Compile composite ant-sca-package.xml package Deploy composite ant-sca-deploy.xml deploy Export MDS ant-sca-deploy.xml exportSharedData Import artifacts to MDS ant-sca-deploy.xml deploy This command demonstrates how to execute ant to deploy a HelloWorld composite to the SOA server. The full path to the deployable JAR file must be specified. Keep in mind that data sources and connection factories for both the 10g and 11g infrastructures can point to the same targets. Furthermore, during the development phase of the project, turn off all services on the 10g infrastructure that performs polling (e.g., JMS Adapter consume, AQ Adapter dequeue, DB Adapter polling). If you do not, polling services will compete on both the old and new infrastructures. Application Server Setup All application server and SOA server configuration that is required for use by your SOA code should be migrated (usually manually) from Oracle Application Server 11g to Oracle WebLogic Server 11g and typically is done before the code migration. All out-of-the-box configurations need not be touched, and only custom configuration, such as custom data sources and connection factories, will need to be created identically in the new environment. Migrate Custom Data Sources and Connection Factories Data sources and connection factories are used by SOA code to connect to a resource or target system (e.g., AQ, JMS queue, database table) without having the need to hardcode passwords within your code. For example, your SOA projects may be leveraging custom created JNDIs that take the following format: jdbc/db/PerfTest eis/db/PerfTest In Oracle WebLogic Server 11g, navigate to the following page to create all custom data sources using the same names and settings as defined in Oracle Application Server 10g: soa_domain ➔ Services ➔ Data Sources Figure 3. Screen displaying the list of data sources in Oracle WebLogic Server 11g Navigate to the following page to create the equivalent connection factories in Oracle WebLogic Server 11g: soa_domain ➔ Deployments ➔ Adapter Name ➔ Configuration ➔ Outbound Connection Pools Create Partitions BPEL domains in 10g are used to group BPEL processes. Each BPEL domain has its own configuration and logging and is created by logging in to the BPEL Admin console at a url similar to the following: http://soa10g:7777/BPELAdmin. There is no concept of BPEL domains in Oracle SOA Suite 11g (actually, there is technically a single BPEL domain). But beginning with Oracle SOA Suite 11g PS2 (11.1.1.3), the concept of partitions is introduced and helps better organize your SOA composites. Partitions, unlike BPEL domains, do not have their own configuration or logging. However, from a code-grouping aspect, they can be treated identically to BPEL domains. For example, if you have a BPEL domain called “Accounting,” simply create a partition called “Accounting.” Thinking of Upgrading to Oracle SOA Suite 11g? continued from page 23
- 27. 4th Qtr 2011 ■ Page 25 Import Shared Schemas and WSDLs If you have shared schemas and/or WSDLs, import them to the MDS, maintaining the same directory structure. If your schemas or WSDLs have imports that are fully qualified references, they must be changed to local references accordingly. Let’s say you have two schemas on your shared server in the following location: http://soa10g:7777/Account/Account.xsd http://soa10g:7777/Custom/CustomAccount.xsd And let’s say that Account.xsd imports the CustomAccount.xsd schema as an HTTP reference as follows: schema import=”http://soa10g:7777/Custom/CustomAccount.xsd” This reference in Account.xsd will need to be changed to a local reference: schema import=”../Custom/CustomAccount.xsd” Finally, zip up the contents of this folder and use ant to import it to the MDS. This would be imported to the MDS under the following location and referenced in your code as such: oramds:/apps/xsd/Account/Account.xsd oramds:/apps/xsd/Custom/CustomAccount.xsd 3. Migrate the Code Migrating SOA Suite 10g code to 11g involves performing pre-migration, migration (either manually through JDeveloper or automatically), followed by various post-migration steps, depending on the functions, logic and references in the code. Migrate the Projects Upgrading BPEL and ESB projects developed originally in JDeveloper 10g involves first going through basic pre-migration steps followed by the actual migration itself, which will migrate the project to an 11g composite. 1. Ensure that projects are migrated in order of dependency. For example, if HelloWorld1 invokes HelloWorld2, then HelloWorld2 must be migrated and deployed to the Oracle SOA Suite 11g server prior to migrating HelloWorld1. 2. Open the HelloWorld1 10g project in JDeveloper 10g. 3. Modify partner link endpoints to point to the 11g service. For example, HelloWorld1 is currently pointing to HelloWorld2 deployed to the old SOA Suite 10g server, referencing it as follows: http://soa10g:7777/orabpel/default/HelloWorld2/HelloWorld2?wsdl. Now that HelloWorld2 is upgraded and deployed to the new SOA Suite 11g server, modify the partner link in HelloWorld1 to point to the newly upgraded HelloWorld2: http://soa11g:8001/soa_infra/services/default/HelloWorld2/HelloWorld2_ ep.wsdl. 4. Save the project in JDeveloper 10g. ant -f ant-sca-deploy.xml deploy -DserverURL=http://soa11g:8001/soa-infra/deployer -Duser=weblogic -Dpassword=welcome1 -DsarLocation=/u01/HelloWorld/deploy/sca_HelloWorld_rev1.0.jar -Dpartition=default -Doverwrite=true -DforceDefault=true -Dconfigplan=/tmp/cfgplan.xml For more information regarding the ant build scripts shipped with Oracle SOA Suite 11g, refer to chapter 43.6.2 “How to Manage SOA Composite Applications with ant Scripts of the Oracle Fusion Middleware Developer’s Guide for Oracle SOA Suite 11g Release 1 (11.1.1)” documentation. Import Artifacts to the MDS It is necessary to import all shared artifacts such as custom DVMs, schemas, WSDLs and fault policies to the MDS prior to migrating the SOA code. In Oracle SOA Suite 10g, the majority of configuration was stored on the file system. This meant that any change to any configuration file must be made on all midtiers. Now, in Oracle SOA Suite 11g, these artifacts are stored centrally in the MDS, which is a database-based repository introduced with Oracle SOA Suite 11g. Migrating Shared Domain Value Maps (DVMs) Although there are numerous ways to migrate DVMs from Oracle SOA Suite 10g to 11g, one approach is detailed in chapter 9.3.7.7 “Upgrading Domain Value Maps (DVMs) and Cross References (XREFs) in the ESB Repository” of the upgrade guide. Migrate BPEL Fault Policies It is possible to maintain multiple fault policies in Oracle SOA Suite 11g. Here I describe, at a high level, how to migrate a single, custom fault policy from 10g to 11g. Simply change the version in fault-bindings.xml from “2.0.1” to “3.0” and change the process element to composite, as highlighted in red below: ?xml version=”1.0” encoding=”UTF-8”? faultPolicyBindings version=”3.0” xmlns=”http://schemas.oracle.com/bpel/faultpolicy” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” composite faultPolicy=”DefaultPolicy”/ /faultPolicyBindings Rename the 10g fault policy file to fault-policies.xml, change the version from “2.0.1” to “3.0”, and add a higher level faultPolicies node, as shown below: ?xml version=”1.0” encoding=”UTF-8”? faultPolicies xmlns=”http://schemas.oracle.com/bpel/faultpolicy” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” faultPolicy version=”3.0” id=”DefaultPolicy” xmlns:env=”http://schemas.xmlsoap.org/soap/envelope/” xmlns:xs=”http://www.w3.org/2001/XMLSchema” xmlns=”http://schemas.oracle.com/bpel/faultpolicy” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” . . . /faultPolicy /faultPolicies continued on page 26
- 28. Page 26 ■ 4th Qtr 2011 Add Fault Policies and Bindings Unlike 10g, there is no concept of a global fault policy in Oracle SOA Suite 11g that will automatically apply to all deployed composites. Therefore, you must manually reference the fault policy and fault binding in composite.xml of each project. 1. Edit composite.xml. 2. Add the following immediately below the /service tag to reference your custom fault policy: property name=”oracle.composite.faultPolicyFile” oramds:/apps/faultPolicies/fault-policy.xml /property property name=”oracle.composite.faultBindingFile” oramds:/apps/faultPolicies/fault-bindings.xml /property Migrate bpel.xml Properties Any custom properties in bpel.xml (10g) will have to be manually added to composite.xml (11g). 1. Open the 10g bpel.xml file, and locate any custom properties. For example: property name=”retryInterval”60/property 2. Open the 11g composite.xml file, and add the 11g version of the property: property name=”jca.retry.interval”60/property Review Mediator Filters The JDeveloper 11g Migration Wizard does not work properly in some cases. For example, when complicated ESB filters are migrated, brackets can be missing which changes the meaning of the filter if a combination of logical operators are used (e.g., AND, OR). In other cases, dollar ($) signs are missing, which results in incorrect XPaths. Review Mediator filters manually after migration and compare them to the ESB filters to ensure correctness. Review and Modify ESB Header Functions Some ESB XPath header functions are made obsolete in 11g and replaced with alternate Mediator XPath functions, shown in Table 3. If the ESB header functions were used for dynamic routing, they will have to be changed. ESB XPath Function (obsolete) Mediator XPath Function ehdr:setOutboundHeader mhdr:setProperty ehdr:getRequestHeader mhdr:getProperty Table 3. Obsoleted ESB XPath functions and their equivalent 11g replacements For example, dynamic routing requires setting the outbound header via the setOutboundHeader function as follows: ehdr:setOutboundHeader(‘/shdr:ESBHeader/shdr:location’,$LocationIn,’shdr=http:// xmlns.oracle.com/esb;’) 5. Open the project in JDeveloper 11g. 6. Navigate through the Migration Wizard. If an error occurs, as shown in Figure 5, the Migration Wizard does not give a specific reason for the failure of the migration. Figure 5. JDeveloper 11g Migration Status In that case, check the logs at (or appropriate directory): c:OracleMiddlewarejdeveloperupgradelogs*.* Post-Migration Steps This is perhaps the most time-consuming step of the entire process because most of the post-migration steps will have to be done manually. After the code has been successfully migrated to 11g, based on the complexity of the project and features used, several post-migration steps may need to be performed. Here, we cover a few of the major ones. Modify DVM Lookups Every DVM lookup must be manually modified. Open all files in the project and locate any DVM lookup functions. Manually modify the DVM function name and the location of the DVM as described. For example, the 10g function may look as follows: orcl:lookup-dvm(“Currency”,”Country”,”US”,”Currency”,”USD”) Both the DVM lookup function and the path to the DVM should be updated accordingly: dvm:lookupValue(“oramds:/apps/dvm/Currency.dvm”,”Country”,”US”,”Currency”,”USD”) Modify Schema Imports Every schema import should be manually modified to use ORAMDS references instead of HTTP references. By doing so, there is no more hardcoding of hostnames and ports. Open all schemas and WSDLs in the project and locate any import or location statements (except for local schemas, which remain unchanged), and convert the HTTP reference to an ORAMDS reference. For example, the 10g import of a schema may look as follows: schema import=”http://soa10g:7777/Account/Account.xsd” The import would be changed to the following: schema import=”oramds:/apps/xsd/Account/Account.xsd” Thinking of Upgrading to Oracle SOA Suite 11g? continued from page 25
- 29. 4th Qtr 2011 ■ Page 27 This approach is recommended because it calls out certain areas that should be completed before starting the code migrating itself, such as developing the deployment process early on, importing all artifacts to the MDS prior to code migration and deciding how you want to create partitions. If you don’t import your DVMs, fault policies and shared schemas to the MDS, it will become a back-and-forth game between the developers (who will constantly complain of missing artifacts in the MDS) and the application administrators. Upgrading from Oracle SOA Suite 10g to 11g is a challenging effort, but it is not an impossible one. Expect to encounter issues no one else has. But by considering some of the points listed here, you may be able to avoid some of the pitfalls the early adopters have faced. C ■ ■ ■ About the Author Ahmed Aboulnaga is a technical director at IPN Web and has extensive experience in Oracle Fusion Middleware technologies. He has an OCE (Oracle SOA Architect) and an OCA (Oracle Application Server 10g) and is a regular speaker at conferences such as OpenWorld, COLLABORATE, IOUG, OAUG and other user groups. He has been involved in the architecting and implementation of large-scale systems involving middleware technologies such as Oracle SOA Suite, Oracle Application Integration Architecture, Oracle Data Integrator, Oracle Integration B2B, Oracle Identity Management and numerous other Oracle middleware technologies. Ahmed can be contacted at ahmed.aboulnaga@ipnweb.com. In 11g, the equivalent would be: mhdr:setProperty(‘out.property.endpointURI’,$LocationIn) Header functions are one of the more problematic areas of the migration. In some cases, such as setting usernames and passwords in the header, these will flat out not work and those functions must be moved to the new “Assign Values” feature in Mediator 11g. Other Considerations The “Oracle Fusion Middleware Upgrade Guide for Oracle SOA Suite, WebCenter, and ADF 11g Release 1 (11.1.1)” has additional steps that may apply to your code that you should read and follow. Test Test, test and test. The underlying code has largely changed, thus; dedicate time to exhaustive unit and end-to-end testing. Typically, developers will rely on client testing tools such as soapUI or the BPEL 10g and Fusion Middleware 11g consoles. Fortunately, identifying and resolving issues uncovered through the compilation and deployment of the JDeveloper 11g SOA projects addresses many of the coding issues, but it is not enough, as there are issues that will only be discovered at runtime. Both SOA Suite 10g and 11g infrastructures should remain available during the upgrade and testing periods. Testing is straightforward: 1. Test payload on your service deployed to SOA Suite 10g. 2. Test payload on your service deployed to SOA Suite 11g. 3. Compare results; they should be identical. Unexpected results or runtime errors are almost always a result of migration problems. We mentioned earlier that the JDeveloper 11g Migration Wizard is somewhat buggy and does not cover all migration scenarios. In this event, manual troubleshooting of the code is required. 4. Install Remaining Environments Follow the “Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite 11g Release 1 (11.1.1)” for installing higher environments such as Test, QA, Staging and Production in a clustered architecture. The application administrators can do this in parallel to the code migration effort by the developers. This is also where a cutover strategy may need to be addressed, as there may be considerations surrounding load balancing and long running processes, some of which are discussed in the “Oracle Fusion Middleware Upgrade Guide for Oracle SOA Suite, WebCenter, and ADF 11g.” 5. Code Promotion Once the higher environments are available and the code migration is complete, follow through your company’s standard code promotion and release management processes. Summary The upgrade approach presented here is an excellent starting point and has aided in ensuring the success of SOA Suite 10g to 11g upgrade projects. Nonetheless, this proposed approach is not meant to replace the existing Oracle documentation, but rather to augment it. USERS GROUP CALENDAR For the most updated calendar, please visit www.ioug.org December 2011 December 13 NYOUG Special Winter Meeting New York Oracle Users Group New Yorker Hotel (212) 978-8890 Email: info@nyoug.org Event URL: http://www.nyoug.org February 2012 February 14-16 RMOUG Training Days Rocky Mountain Oracle Users Group Colorado Convention Center, Denver Event URL: http://www.teamycc.com/ RMOUG_2012_Conference/ Registration.html April 2012 April 22-26 COLLABORATE 12 – IOUG • Forum 2012 Mandalay Bay Convention Center, Las Vegas Event URL: http://events.ioug.org/p/ cm/ld/fid=28
- 30. Page 28 ■ 4th Qtr 2011 Beating the Optimizer By Jonathan Lewis H ow do you access data efficiently if there’s no perfect index? Sometimes the only answer is to create the index you need; however, there are times when you can rewrite the query to be “efficient enough” by taking advantage of the indexes you already have. This article will examine methods for writing SQL that explicitly combine indexes in ways that the optimizer cannot yet manage and, by doing so, minimize the number of accessible table blocks. The Problem In a previous issue of SELECT Journal, I discussed a number of options for optimizing a two-table join, including a strategy for turning the join into a three-table join in a way that effectively allowed Oracle to treat an index on a table as if it were totally independent. In this issue, I want to take a step back and talk about optimizing access to a single table using the same type of strategy. Listing 1 is a script to generate a data set that I’m going to use in this article to describe the type of problem I want to address. To build this data set, you will need approximately 200MB of free space for the table and its indexes. For reference, I created the demonstration data in a locally managed tablespace with an 8KB block size and 1MB uniform extents, using freelist management rather than ASSM. create table big_table as with generator as ( select /* + materialize */ rownum id from dual connect by rownum = 10000 ) select trunc(dbms_random.value(1,500)) col1, trunc(dbms_random.value(1,500)) col2, trunc(dbms_random.value(1,500)) colA, trunc(dbms_random.value(1,500)) colB, rownum n1, rpad(‘x’,100) padding from generator v1, generator v2 where rownum = 1000000; create index bt_i1 on big_table(col1, col2) compress 1; create index bt_i2 on big_table(colA, colB) compress 1; Listing 1: Sample data set. A couple of indexes on the table will offer high precision for queries with predicates such as col1 = 99 and col2 = 499, or colA = 1 and colB = 100. Thanks to the magic of btree-bitmap conversion, the indexes may even be effective for predicates such as col1 = 99 and colA = 1 (Listing 2. shows an edited execution plan for this last example): -------------------------------------------------------------- | Id | Operation | Name | Rows | -------------------------------------------------------------- | 0 | SELECT STATEMENT | | 4 | | 1 | TABLE ACCESS BY INDEX ROWID | BIG_TABLE | 4 | | 2 | BITMAP CONVERSION TO ROWIDS | | | | 3 | BITMAP AND | | | | 4 | BITMAP CONVERSION FROM ROWIDS| | | | 5 | SORT ORDER BY | | | |* 6 | INDEX RANGE SCAN | BT_I1 | 2000 | | 7 | BITMAP CONVERSION FROM ROWIDS| | | | 8 | SORT ORDER BY | | | |* 9 | INDEX RANGE SCAN | BT_I2 | 2000 | -------------------------------------------------------------- Listing 2: Example of btree-bitmap conversion plan If you’re running 10g Standard Edition, though, you won’t get much help from this set of indexes for the predicate col1 = 99 and colA = 1 because Standard Edition does not support bitmap indexes — which means it can’t do btree-bitmap conversion (and the and_equal execution path, which might have been a viable alternative, has been deprecated in 10g) — so you might expect to see a table scan. Worse still, there aren’t any indexes that will help a predicate like col2 = 99 and colB = 499 — although, in different circumstances, with different data patterns, an index skip scan on one of the indexes might be reasonably effective. Options If the col2/colB query isn’t run very often and doesn’t have to give a rapid result, you may decide you don’t mind if it is inefficient and just leave things as they are. However, if this query is important and will be run frequently, you may want to do something about it. You could simply build an index on (col2, colB) so that the optimizer could find exactly the right table rows very precisely, but that’s another index to maintain as you modify the data, another chunk of disc space used, and another object encroaching on your buffer cache. Also, every time you add an index to solve one query performance problem, you may affect many other queries that reference the same table. Can the problem be addressed by rewriting the query? It’s actually very easy to do once you got past one mental block: indexes are tables. Specifically, an index is very much like a table defined by the statement of the form: select {index_columns}, rowid from table order by {index_columns}. Start by taking a leaf out of the optimizer’s book — the one about the ways that Oracle can combine indexes. Before 10g, you used to have the and_equal execution path, and you still have the index_join path. Both paths manipulate the contents of multiple indexes to minimize the number of visits to the underlying table — and, in many cases, it’s the number and randomness of visits to the table that is the major component of the work done by a query. The and_equal path applied only to single column, nonunique indexes where the predicates were all equality, and Oracle could combine up to 5 separate indexes on the same table to derive the smallest set of rowids it needed before visiting the table. Listing 3 shows a query on a sample dataset with a hinted and_equal plan:
- 31. 4th Qtr 2011 ■ Page 29 in a live system, it’s quite possible that my two indexes would be very well cached while most of my table would have to be read from disc — so the saving could be much more than just a reduction in block counts. Fortunately, there is a strategy that I can use: explictly coding a transformation that Oracle cannot generate internally. Listing 4 shows a slightly unusual piece of SQL that allows Oracle to take the path I want, doing something similar to the and_equal after a couple of index fast full scans. select big_table.* from (select rowid rid from big_table where col2 = 99 order by rowid) rid1, (select rowid rid from big_table where colB = 499 order by rowid) rid2, big_table where rid2.rid = rid1.rid and big_table.rowid = rid2.rid; --------------------------------------------------------- | Id | Operation | Name | Rows | --------------------------------------------------------- | 0 | SELECT STATEMENT | | 4 | | 1 | NESTED LOOPS | | 4 | | 2 | MERGE JOIN | | 4 | | 3 | VIEW | | 1992 | | 4 | SORT ORDER BY | | 1992 | |* 5 | INDEX FAST FULL SCAN | BT_I1 | 1992 | |* 6 | SORT JOIN | | 1992 | |* 7 | INDEX FAST FULL SCAN | BT_I2 | 1992 | | 8 | TABLE ACCESS BY USER ROWID| BIG_TABLE | 1 | --------------------------------------------------------- Listing 4 – Query and plan of a “manual and_equal”. I got lucky with my plan because Oracle picked the join order I wanted and then did merge join at the right place. If I had wanted to be sure that this execution plan appeared in a production system, I would have included about a dozen hints, including a leading() hint, a couple of index_ffs() hints and a pair of no_eliminate_oby() hints. There are alternatives to this plan. Although I started by emulating the and_equal, I remembered that Oracle also has an index_join path as another way of combining information from multiple indexes — this time using a hash join. Once I’ve got the basic idea of writing inline views that make an index act like a spare table, I can use any index access path and any join method I like — so Listing 5 is another version of the query, including the hints this time, using a hash join instead of a merge join. select /*+ qb_name(main) leading(rid1@r1, rid2@r2, big_table(@main) use_hash(rid2@r2) no_swap_join_inputs(rid2@r2) use_nl(big_table@main) rowid(big_table@main) index_ffs(@r1 big_table@r1(col1, col2)) index_ffs(@r2 big_table@r2(colA, colB)) */ big_table.* from (select /*+ qb_name(r1) */ rowid rid from big_table where col2 = 99) rid1, (select /*+ qb_name(r2) */ rowid rid from big_table where colB = 499) rid2, big_table where rid2.rid = rid1.rid and big_table.rowid = rid2.rid ; select /*+ and_equal(big_table bt_1 bt_a) */ * from big_table where col1 = 99 and colA = 1; --------------------------------------------------------- | Id | Operation | Name | Rows | --------------------------------------------------------- | 0 | SELECT STATEMENT | | 4 | |* 1 | TABLE ACCESS BY INDEX ROWID| BIG_TABLE | 4 | | 2 | AND-EQUAL | | | |* 3 | INDEX RANGE SCAN | BT_1 | 2000 | |* 4 | INDEX RANGE SCAN | BT_A | 2000 | --------------------------------------------------------- Listing 3: Example of an and_equal execution path To make this plan appear with the sample data set, I had to create two new indexes on (col1) and (colA), respectively, and then hint the optimizer into using the correct indexes in the correct way. What the optimizer has done here is a range scan on the two indexes. Because each scan was on a simple equality, and because of the way nonunique indexes append the rowid to the index entry before sorting and storing the entries, each range scan automatically produces a sorted list of rowids, which allows Oracle to do a merge join of maximum efficiency between the two sets of rowids. Any rowid that appears in both sets is a rowid for which both predicates are true, so the merge join identifies exactly those rows you need from the table in the cheapest possible way. (With compression, the two indexes were about 1,540 leaf blocks each, which means you scanned about eight leaf blocks in total and then visited four table blocks when the best you could have done with a single index would have been to visit 2,000 separate table blocks.) But, as I pointed out, the and_equal path has been deprecated in 10g. It is, after all, limited to single column indexes and equality predicates while the btree-bitmap conversion has no such limitation. If you go back to Listing 2, you can see the plan I got without hinting the and_equal on my Enterprise Edition version of Oracle. (Remember, this won’t work on Standard Edition because bitmap indexes are not available.) The btree-bitmap conversion mechanism works as follows: Do a range scan of each index and collect the rowids as you go. Because it’s a range scan on a multicolumn index, there is no guarantee of getting sorted rowids; so Oracle sorts each set of rowids into order. Rather than doing a merge join, Oracle says (in effect): imagine listing the table in physical rowid order. For each rowid in my list, put a 1 beside the corresponding row, and a 0 beside every other row. This produces a string of bits that can be used with a bitmap and. So Oracle constructs two string of bits, combines them, then reverses the rowid/bitmap conversion to get the list of rowids of rows you want. (The conversion actually works by doing arithmetic on a model of 736 rows in a block — depending on block size and various other details — unless you’ve set the Hakan factor by using the minimize_records_per_block option.) Oracle minimizes trips to the table by combining information from indexes, but how does this help with the problem? My problem predicate is about the second column in two indexes — so neither plan is possible — but it would be nice to find a strategy that is similar because my test table totaled 17,544 blocks while my two indexes cover 2,072 blocks each. If I could find a way of scanning only 4,144 blocks to visit the four rows I want from the table, that’s an improvement over a full tablescan. When comparing the numbers, it’s also worth remembering that, continued on page 33
- 32. Page 30 ■ 4th Qtr 2011 Tips, Notes and Information from InSync 2011 By Tony Jambu T he Australian InSync 2011 conference was held in Sydney in August. The format is similar to our COLLABORATE conference in which the Australian OUG and OAUG had a combined conference. Below are some of the notes and tips from the conference and various sources. Best InSync 2011 Presentation by Connor McDonald Of all the presentations that I attended, the best presentation was from Connor McDonald, Oracle ACE director and longtime OakTable member. The presentation was titled “Managing Optimiser Statistics — A Better Way.” It was the most entertaining and educational presentation I saw. The presentation consisted of 360 slides. Yes, 360 slides! That’s not a typo. If anyone can present 360 slides in 45 minutes and still get the message across, Connor can. If you are interested, the presentation can be read or downloaded from the following link: http://www.slideshare.net/InSync2011/databse-technology-2-connor- mcdonald-managing-optimiser-statistics-a-better-waypdf Here are just a few of his tips: 1) Statistics By Stealth In Oracle 10g, the automatic system statistics gathering is carried out via the GATHER_STATS_JOB job. SQL select owner, job_name, enabled 2 from dba_scheduler_jobs 3 where owner = ‘SYS’; OWNER JOB_NAME ENABLED ---------------------------------------------------- SYS PURGE_LOG TRUE SYS FGR$AUTOPURGE_JOB TRUE SYS GATHER_STATS_JOB FALSE SYS AUTO_SPACE_ADVISOR_JOB TRUE Running the same query in 11g produces SQL select owner, job_name, enabled 2 from dba_scheduler_jobs 3 where owner = ‘SYS’; OWNER JOB_NAME ENABLED ---------------------------------------------------- . . SYS AUTO_SPACE_ADVISOR_JOB FALSE SYS GATHER_STATS_JOB FALSE SYS PURGE_LOG TRUE . Here is where one would think that the system will not gather statistics on a daily basis. The reality is that the default behavior is that the statistics will still be gathered. As Connor puts it, it is being gathered in “super stealth mode.” SQL select client_name, status 2 from DBA_AUTOTASK_CLIENT; CLIENT_NAME STATUS ------------------------------------ -------- auto optimizer stats collection ENABLED auto space advisor ENABLED sqltuning advisor ENABLED These tasks are known as the “autotasks.” In 11g, Oracle introduced these sets of autotasks. Further information can be obtained here: http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ ID:1133388300346992024 http://download.oracle.com/docs/cd/B28359_01/server.111/b28318/ mgmt_db.htm#CNCPT1375 To turn off the automatic statistics gathering in 11g, you will need to call the DBMS_AUTO_TASK_ADMIN package: BEGIN DBMS_AUTO_TASK_ADMIN.DISABLE( client_name = ‘auto optimizer stats collection’, operation = NULL, window_name = NULL); END; / 2) Histograms When gathering table statistics, it is possible to explicitly specify which columns should have histograms statistics gathered. This would be tedious, as it would require you to code the column name every time. With Oracle 11g, it is now simplified with the DBMS_STATS.SET_SCHEMA_PREFS and DBMS_STATS.SET__PREFS. The SET_[SCHEMA|TABLE]_PREFS procedures allows you to set default values that are used by the DBMS_STATS.GATHER_[SCHEMA|TABLE]_STATS. As an example, you could set all tables to not gather any histogram statistics with: EXEC DBMS_STATS.SET_SCHEMA_PREFS(‘SCOTT’, ‘METHOD_OPT’, ‘for all columns size 1’); SELECT Star
- 33. 4th Qtr 2011 ■ Page 31 3 where d.deptno= e.deptno 4 and e.jobid= 1 5 and e.salary 5000; no rows selected -------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | -------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 1 | HASH JOIN | | 89 | 2136 | | 2 | TABLE ACCESS FULL | DEPT | 1 | 10 | |* 5 | TABLE ACCESS FULL | EMP | 4444 | 62216 | -------------------------------------------------------------- Execution Plan Note ----- - cardinality feedback used for this statement Notice that the execution plan above informs us that it had used the cardinality feedback to improve its execution plan. Another way is to query the USE_FEEDBACK_STATS column from V$SQL_SHARED_CURSOR view. A nice enhancement to the Oracle optimizer would be for it to learn on the fly. If the expected rows or expected CPU/time did not match the actuals, it would stop and re-evaluate its plan with the new information and re-parse and re-run the statement automatically. This feature is currently available in other vendor’s optimizer. IGNORE_ROW_ON_DUPKEY_INDEX Hint Another well-known Oracle ACE Director and OakTable member presenting at InSync 2011 was Richard Foote. His presentation was called “10 Things You Might Not Know About Oracle Indexes.” There were lots of interesting facts of which I was unaware. The presentation had many examples and demos to help the audience comprehend the material being covered. http://www.slideshare.net/InSync2011/database-technology-2-richard- foote-10-things-you-probably-dont-know-about-oracles-indexespdf One interesting piece of information was the topic on IGNORE_ROW_ON_ DUPKEY_INDEX hint. This hint was introduced in Oracle 11gR2. http://download.oracle.com/docs/cd/E11882_01/server.112/e22487/ chapter1.htm#FEATURENO08842 http://download.oracle.com/docs/cd/E11882_01/server.112/e26088/ sql_elements006.htm#SQLRF30052 This is intended to assist with large INSERTs that may have duplicate records as defined by the Unique Key. When inserting records without this hint specified, any records that are duplicates will cause an ORA-00001 – unique constraint error followed by rollback of all inserted rows. However, with the IGNORE_ROW_ON_DUPKEY_INDEX hint specified, any duplicate key rows will be ignored (with no warning) while the rest of the rows will be accepted and inserted in to the table. The syntax for using the hint is: Followed by explicitly specifying which columns require histograms with: EXEC DBMS_STATS.SET_TABLE_PREFS(‘SCOTT’, ‘EMP’, ‘METHOD_OPT’, ‘for column SAL size 25’); EXEC DBMS_STATS.SET_TABLE_PREFS(‘SCOTT’, ‘EMP’, ‘METHOD_OPT’, ‘for column DEMPNO size 10’); The next time you execute DBMS_STATS.GATHER_[SCHEMA|TABLE]_STATS for Scott’s schema or EMP table, the SAL and DEPNO columns would gather histogram statistics. 3) Oracle Cardinality Feedback The Oracle optimizer is only as good as the input information or statistics that it has. Sometimes it gets things wrong — especially when its estimated cardinality (e-Rows) differs significantly from the actual rows (A-Rows) retrieved. SELECT /*+ gather_plan_statistics */ COUNT(1) cnt, SUM(S.QUANTITY_SOLD) qty_sold, SUM(C.UNIT_COST) sum_cost FROM sh.sales S, sh.costs c WHERE S.PROD_ID = C.PROD_ID AND s.time_id = :from_date ------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | ------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:04.40 | | 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:04.40 | |* 2 | HASH JOIN | | 1 | 12M| 17M|00:00:03.45 | | 3 | PARTITION RANGE ITERATOR | | 1 | 10699 | 15248 |00:00:00.01 | | 4 | TABLE ACCESS BY LOCAL INDEX ROWID| SALES | 9 | 10699 | 15248 |00:00:00.01 | | 5 | BITMAP CONVERSION TO ROWIDS | | 9 | | 15248 |00:00:00.01 | |* 6 | BITMAP INDEX RANGE SCAN | SALES_TIME_BIX | 9 | | 17 |00:00:00.01 | | 7 | PARTITION RANGE ALL | | 1 | 82112 | 82112 |00:00:00.06 | | 8 | TABLE ACCESS FULL | COSTS | 28 | 82112 | 82112 |00:00:00.04 | ------------------------------------------------------------------------------------------------------- But can Oracle learn from its mistake? With Oracle 11gR2, it can with its Cardinality Feedback feature. The following example is from Connor’s presentation. The first query was execute and its plan is shown below. SQL select e.empno, d.dname 2 from empe, dept d 3 where d.deptno= e.deptno 4 and e.jobid= 1 5 and e.salary 5000; no rows selected -------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | -------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 1 | MERGE JOIN | | 4444 | 104K | | 2 | TABLE ACCESS BY INDEX ROWID| DEPT | 100 | 1000 | | 3 | INDEX FULL SCAN | DEPT_IX | 100 | | |* 4 | SORT JOIN | | 4444 | 62216 | |* 5 | TABLE ACCESS FULL | EMP | 4444 | 62216 | -------------------------------------------------------------- With Cardinality Feedback working and learning that its actual rows did not match its estimated cardinality, it would “remember” the actual values. If you re-ran the same query with no changes, you would get: SQL select e.empno, d.dname 2 from empe, dept d continued on page 32
- 34. Page 32 ■ 4th Qtr 2011 The test he carried out inserted about 600,000 rows into a copy of the SALES table. About half of the rows already existed. The three approaches were: 1. Issue a select to see if there is already a matching value and only insert if there is not. 2. Issue an INSERT and use an EXCEPTION handler to catch any duplicate values. 3. Use the IGNORE_ROW_ON_DUPKEY_INDEX hint. Here are the elapsed times for each approach: IGNORE_ROW_ON_DUPKEY_INDEX Exception Handler Check before insert 0 50 100 150 200 250 300 350 Elapsed Time (s) ■ SELECT ■ INSERT ■ COMMIT ■ ROLLBACK The IGNORE_ROW_ON_DUPKEY_INDEX method is about 20 percent faster than using the exception approach. The surprise was that the manual method of checking the existence of duplicate records first before inserting was the fastest and most efficient by a mile. There are limitations for the use of this hint, which include the following points: •• If you specify index, then the index must exist and be unique. Otherwise, the statement causes ORA-38913. •• You must specify exactly one index. •• You can specify either a CHANGE_DUPKEY_ERROR_INDEX or IGNORE_ROW_ ON_DUPKEY_INDEX hint in an INSERT statement, but not both. •• It only applies to single-table INSERTs. •• It is not supported with UPDATE or MERGE. However, the most important limitation is that this hint disables both the APPEND operation mode and parallel DML. Oracle’s Exadata Smart Flash Logging Beginning with Exadata Storage 11.2.2.4.0 is a new feature called Exadata Smart Flash Logging. This feature speeds database log I/O. It achieves this by utilizing a small proportion of the storage’s flash storage for redo writes. When a redo write is required, it will initiate two simultaneous redo writes: one to the traditional redo file system and the other to the flash memory. The redo write operation is considered complete when either of the two operations is completed first. This provides increased I/O performance and throughput. Here is the example Richard Foote used to demonstrate the feature: SQL create table radiohead (id number constraint radiohead_pk_i primary key using index (create unique index radiohead_pk_i on radiohead(id)), name varchar2(20)); Table created. SQL select index_name, uniqueness, table_name from dba_indexes where index_name = ‘RADIOHEAD_PK_I’; INDEX_NAME UNIQUENES TABLE_NAME ------------------------------ --------- ------------------------------ RADIOHEAD_PK_I UNIQUE RADIOHEAD SQL insert into radiohead select rownum, ‘OK COMPUTER’ from dual connect by level = 10; 10 rows created. SQL commit; Commit complete. SQL insert into radiohead select rownum, ‘OK COMPUTER’ from dual connect by level = 12; insert into radiohead select rownum, ‘OK COMPUTER’ from dual connect by level = 12 * ERROR at line 1: ORA--00001: unique constraint (BOWIE.RADIOHEAD_PK_I) violated We now introduce the hint and insert duplicate records as a test. insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(radiohead,radiohead_pk_i) */ into radiohead select rownum, ‘OK COMPUTER’ from dual connect by level = 12; 2 rows created. SQL insert /*+ IGNORE_ROW_ON_DUPKEY_INDEX(radiohead(id)) */ into radiohead select rownum, ‘OK COMPUTER’ from dual connect by level = 13; 1 row created. SQL commit; Commit complete. SQL select * from radiohead; ID NAME ---------- -------------------- 1 OK COMPUTER 2 OK COMPUTER 3 OK COMPUTER 4 OK COMPUTER 5 OK COMPUTER 6 OK COMPUTER 7 OK COMPUTER 8 OK COMPUTER 9 OK COMPUTER 10 OK COMPUTER 11 OK COMPUTER 12 OK COMPUTER 13 OK COMPUTER 13 rows selected. What about its performance implication? Guy Harrison, director of research and development at Quest and author of various Oracle performances books, has done some work on this. SELECT Star continued from page 31
- 35. 4th Qtr 2011 ■ Page 33 From Oracle’s white paper titled “Exadata Smart Flash Cache Features and the Oracle Exadata Database Machine”: http://www.oracle.com/technetwork/ database/exadata/exadata-smart-flash-cache-366203.pdf Smart Flash Logging works as follows. When receiving a redo log write request, Exadata will do parallel writes to the on-disk redo logs as well as a small amount of space reserved in the flash hardware. The database is immediately notified of completion. If the disk drives hosting the logs experience slow response times, then the Exadata Smart Flash Cache will provide a faster log write response time. Conversely, if the Exadata Smart Flash Cache is temporarily experiencing slow response times, then the disk drive will provide a faster response time. Given the speed advantage the Exadata flash hardware has over disk drives, log writes should be written to Exadata Smart Flash Cache, resulting in very fast redo write performance. This algorithm will significantly smooth out redo write response times and provide overall better database performance. SELECT Star Mailing List For those who are already on the SELECT Star mailing list, most of the above information may be old news to you. For those of you that would like to get the latest news as soon as it occurs, why not subscribe? It contains information that is not published in this journal. Subscribe to the mailing list by visiting http://groups.yahoo.com/group/ Select_Star/ or emailing Select_Star-subscribe@yahoogroups.com. ■ ■ ■ About the Author Tony Jambu has more than 19 years of Oracle experience as a development DBA and database architect. Jambu has an in-depth working knowledge of Oracle and its breadth of technology and unreservedly shares his knowledge with others by presenting regularly both internationally and in Australia, winning best presentations at various conferences. He has been the author of SELECT Star column for IOUG’s SELECT Journal since the magazine’s inception in 1993. Jambu was awarded the “Oracle Consultant of the Year” by Oracle Magazine and “Oracle ACE” by Oracle Corporation for his knowledge and experience in Oracle technology as well as for his technical contribution to the Oracle community. Advertisers’ Index The International Oracle Users Group and SELECT Journal would like to thank the following members of the community for their support. For more information about their products and services, visit their Web sites. And be sure to let them know you saw them in SELECT Journal! Advertiser . . . . . . . . . . . . . . . . . Web Address . . . . . . . . . . . . . . Page Number COLLABORATE 12 . . . . . . . . . . . . . . . . http://collaborate12.ioug.org . . . . . . . . Inside Front Cover embarcadero . . . . . . . . . . . . . . . . . . . . www.embarcadero.com . . . . . . . . . . . . 17 IBM . . . . . . . . . . . . . . . . . . . . . . . . . . www.ibm.com . . . . . . . . . . . . . . . . . . . 4 --------------------------------------------------------- | Id | Operation | Name | Rows | --------------------------------------------------------- | 0 | SELECT STATEMENT | | 4 | | 1 | NESTED LOOPS | | 4 | |* 2 | HASH JOIN | | 4 | |* 3 | INDEX FAST FULL SCAN | BT_I1 | 1992 | |* 4 | INDEX FAST FULL SCAN | BT_I2 | 1992 | | 5 | TABLE ACCESS BY USER ROWID| BIG_TABLE | 1 | --------------------------------------------------------- Listing 5: The query with a hash join between indexes. Warning Once you start writing code like this, you have to remember that it is dependent on hinting and on the existence of very specific indexes. You must catalog and document any examples of this strategy very carefully. Occasionally, Oracle changes the way hints work, or introduces a new feature that requires you to modify the set of hints you’ve used. People often drop indexes or change index definitions. In either case, you need to check your special code before any such changes go into production. Conclusion Once you fully appreciate that an index can be treated as if it were just another form of a table, you may find ways of writing code that emulates transformations to combine indexes in ways that the cost-based optimizer isn’t yet able to do, thus making queries “efficient enough” without having to create new indexes C ■ ■ ■ About the Author Jonathan Lewis is a freelance consultant whose experience with Oracle goes back more than 23 years. He specializes in physical database design, the strategic use of the Oracle database engine and solving performance issues. Lewis is the author of “Cost Based Oracle – Fundamentals” and “PracticalOracle 8i – Designing Efficient Databases” and has contributed to three other books about Oracle. Further details of his published papers, presentations and tutorials can be found through his blog at http://jonathanlewis.wordpress.com and at his older website at http://www.jlcomp.demon.co.uk. Beating the Optimizer continued from page 29 BI Tip | Enterprise Manager vs. WebLogic Administration Console With OBIEE 11g, you may be confused by having two web-based administration interfaces. The easy way to remember their use is that for most tasks, including starting up, shutting down and administering components, you use Enterprise Manager; the only thing you really need to use the WebLogic Administration Console is for maintaining the users and groups in the WebLogic LDAP server. From Mark Rittman’s COLLABORATE 11 presentation “Oracle Business Intelligence 11g Architecture and Internals”
- 36. Page 34 ■ 4th Qtr 2011 ”Quick Study” COLLABORATE 11 Bootcamp Leaders The Questions: ▶ ▶ ▶ ▶ ▶ 1. Best tip/presentation learned from COLLABORATE 2011? ▶ ▶ ▶ ▶ ▶ 2. Dumbest thing you ever did working in IT? ▶ ▶ ▶ ▶ ▶ 3. Best geek swag you’ve acquired? ▶ ▶ ▶ ▶ ▶ 4. What do you want to see covered at COLLABORATE? ▶ ▶ ▶ ▶ ▶ 5. How have you contributed to the IOUG? ▶ ▶ ▶ ▶ ▶ 6. How do you stay connected? Ian Abramson – Business Intelligence 1.The best was the “Ask Tom” session where Tom Kyte discussed the Exadata platform. He explained how to achieve top performance with the Exadata machine and how migrating applications from traditional hardware to Exadata can be done to achieve the expected performance gains. 3.The best swag I ever got at a conference was a BMW Z3! That’s right. I won a car at the Oracle conference in LA. 4.The best presentations are the ones that illustrate how people created their solutions. COLLABORATE is a place I often can learn from others. In addition, I really like to see how people manage large volumes of data. 5.I have been a longtime member of the IOUG. I go back to the early ’90s when I first joined. I attend the IOUG Live events and then volunteered for the conference committee. After the conference committee, I worked my way through various volunteer roles until I was elected to the IOUG Board of Directors. I then had the privilege of serving as the IOUG president for two years. It was a great time of change at Oracle, and I had the chance to be part of it. 6.My most important connections is my network of Oracle and data friends. I try to regularly connect and see how things are going. You can find me at http://ians-oracle.blogspot.com/ — there I discuss all things Oracle, Big Data, data warehousing, sports or any other topic that I might find blog-worthy. Steve Lemme – Manageability 1.One of the last conference sessions — and one of the best in case you missed it — was Oracle Enterprise Manager Grid Control Deployment Best Practices by Werner De Gruyter of Oracle. 2.Arizona does not observe daylight saving time, and when I moved the production systems to be on automated network time, the wrong time zone was selected and factory production had to be stopped until that future time was current. 3.A Platinum Technologies jacket that still fits. 4.Cloud and 360 degree Oracle Manageability — Applications, Database, Middleware, Network, Systems and Storage. 5.I have been an attendee since IOUW, a volunteer, presenter and board member. 6.Oracle OpenWorld, COLLABORATE, Oracle and User Group social networks, email and old fashion phone calls to contacts I have met over the past 12 years in the Oracle community. Mark Rittman 1.The best presentation, from my perspective, was a very candid update on the Oracle BI product roadmap from Balaji Yelamanchili. There were lots of questions from the audience and discussions on where we wanted the products to be going. 2.Filling a test customer database with “humorous” customer names, then not realizing they were going to demo the database to senior management without telling me. 3.Probably the range of tech goodies we can pick from as speakers at the UKOUG event. 4.Anything more technical on Oracle BI, DW and EPM. Many sessions I see at conferences are beginners-only, and I’m keen to see anything on OBIEE architecture, performance tuning, internals, repository design and topics like that. 5.I’ve presented at COLLABORATE for about five years now, including running the BI Boot Camp at last year’s event in Orlando. 6.I’ve got the lot — an iPad, iPhone, Macbook Air and lots of WiFi, 3G connections. In terms of social media, I’m on Twitter (@markrittman), Facebook and, of course, our blog at http://www.rittmanmead.com. Michael Rosenblum – Database Development 1.Robert Freeman’s presentation about Oracle’s metadata was a pivotal part of my Database Development Boot Camp. I also really appreciated his approach to the DBA’s new role in the contemporary IT world (aka “DBA 3.0”). 2.On several occasions, I grabbed a laptop to run some overnight tasks from home and forgot the charger. Of course, you always hope that the battery will last long enough to complete the job. But Murphy’s Law is always in effect — both times I’ve had major issues and had to drive to the office around 1 a.m. to finish what I was doing. 3.A strange combination of pen/laser pointer/flashlight/ USB drive in the same body (got it from NYOUG as a speaker gift).
- 37. 4th Qtr 2011 ■ Page 35 2.On his last day on the job, my predecessor mentioned that it’s a best practice to change all the passwords when a DBA leaves the company. I changed SYS and SYSTEM passwords. Good DBA. I changed Unix passwords for the oracle user. Clever fellow. I changed all the role passwords on the ERP database without knowing the old passwords. Wait, I did what? 3.I won a drawing for a Kindle at COLLABORATE 2009. 4.OEM is Oracle’s answer to unified, simplified systems management. A new release is coming out this fall, and we need to provide as much information as we can to help attendees get the most out of this powerful tool. 5.I’ve been a speaker at several COLLABORATE conferences and was proud to represent IOUG speaking at OpenWorld. The last three years, I’ve reviewed conference presentations and white papers and was a Boot Camp co-manager last year. I volunteered for next year’s conference committee, proving once again that I have more spare time than sense. 6.I subscribe to several technology newsletters and keep an eye on LinkedIn for current trends and updates. 4.I would like to see anything that crosses department borderlines. I understand that deep-dive technical presentations are the main reason that people attend conferences, but there should also be some time to lift your head from the keyboard and ask “why?” Also, having different viewpoints presented always helps to broaden one’s horizons. 5.I have contributed directly to the IOUG by writing articles for the SELECT Journal/tip booklets and by presenting annually at COLLABORATE starting in 2006. I have contributed in a less tangible way by trying to bridge the gap between DBAs/developers/system architects by promoting overlapping topics, open discussions, etc. 6.I stayed out of social networks as long as I could, but have finally been pulled into LinkedIn and Facebook and have started my own blog. Although, the good old Oracle-L mailing list is still as constant for me as my morning cup of coffee. Ray Smith – Virtualization 1.I learned that it is not only possible, but relatively easy to virtualize Oracle database on VMware. Just like any other technology stretch, Oracle on VMware is possible if you pay attention to the details. Submit an Article to IOUG SELECT Journal is IOUG’s Quarterly Publication We are always looking for new authors and articles for 2012. Interested in submitting an article? Visit www.ioug.org/selectjournal/ submit.cfm for more information. Questions? Contact SELECT Journal Managing Editor Theresa Wojtalewicz at (312) 673-5870, or email her at twojtalewicz@ioug.org. IOUG Is Looking for New Materials for the 2012 Best Practices Booklet Submissions should be 500-1,000 words long; due to space constraints, we ask that your submission have a specific focus as opposed to any overarching database principles. Tips can range from beginning- to advanced-level skills and should include the actual code and queries used (screenshots and other small graphics are also acceptable). If you have any questions about this project, please contact our Best Practices Booklet Managing Editor Theresa Wojtalewicz, at (312) 673-5870, or email her at twojtalewicz@ioug.org.PracticesTips Best A Compilation of Technical Tips from the Independent Oracle Users Group www.ioug.org Sixth Edition Booklet IOUG