Deploying oracle rac 10g with asm on rhel and sles with svc

Deploying Oracle RAC 10g with ASM on
Red Hat Enterprise Linux and SUSE
Linux Enterprise Server
Using IBM System Storage SAN Volume Controller

Betty Mason, Bob Gonzalez

IBM Systems and Technology Group
Open Systems Lab, San Jose, California

September 2008

© Copyright IBM Corporation, 2008. All Rights Reserved.
All trademarks or registered trademarks mentioned herein are the property of their respective holders

Table of contents
Abstract........................................................................................................................................1
Executive summary ....................................................................................................................1
Introduction .................................................................................................................................2
IBM System Storage SAN Volume Controller ......................................................................................... 2
Replication services ................................................................................................................................. 3
FlashCopy ......................................................................................................................... 3
Metro Mirror and Global Mirror .......................................................................................... 3
Device Mapper Multipath ......................................................................................................................... 3
Oracle technologies....................................................................................................................4
Oracle RAC 10g Release 2 and Oracle Clusterware .............................................................................. 4
Oracle Cluster Ready Services (CRS)..................................................................................................... 4
Oracle Cluster Synchronization Services ................................................................................................ 5
Establish device persistence with Oracle ASMLib................................................................................... 5
Oracle Automatic Storage Management (ASM) ...................................................................................... 5
Advantages of ASM technology ........................................................................................ 6
ASM disk groups ............................................................................................................... 6
ASM failure groups ............................................................................................................ 6
Lab test environment..................................................................................................................7
Oracle RAC servers ................................................................................................................................. 7
Storage hardware .................................................................................................................................... 8
Topology of four-node Oracle RAC 10g Release 2 with ASM................................................................. 8
Storage configuration by example ............................................................................................9
Storage configuration on SVC ................................................................................................................. 9
Understanding considerations for SVC configuration ....................................................... 9
Using SVC CLI to configure storage ................................................................................. 9
Creating virtualized storage from storage arrays ............................................................ 10
Creating host connections............................................................................................... 11
Configuration on server nodes............................................................................................................... 13
Configuring DeviceMapper Multipath for mpio connections............................................ 13
Identifying storage on cluster nodes (SVC Vdisks for ASM)........................................... 14
Best-practice recommendations .............................................................................................15
Summary....................................................................................................................................15
Resources..................................................................................................................................16
Trademarks and special notices..............................................................................................17

Deploying Oracle RAC 10g with ASM on Red Hat Linux and SUSE Linux with IBM SAN Volume Controller. Ver: Sept 11, 2008
http://www.ibm.com/support/techdocs
© Copyright IBM Corporation, 2008

Abstract
This white paper details the configuration of the IBM System Storage SAN Volume Controller
(SVC) in a software environment consisting of Oracle Database 10g with Oracle Real Application
Clusters (RAC) and Oracle Automatic Storage Manager (ASM) on a four-node cluster running Red
Hat Enterprise Linux or SUSE Linux Enterprise Server operating systems.

Executive summary
This white paper details the step-by-step configuration of the IBM® System Storage™ SAN Volume
Controller (SVC) with Oracle Real Application Clusters 10g Release 2 (RAC) and Oracle Database 10g
Automatic Storage Manager (ASM) on the Red Hat Enterprise Linux 4 and SUSE Linux Enterprise Server 9
operating systems. These configuration guidelines may be applicable to newer versions of these operating
systems. The test exercise demonstrates that it is easy to configure the SVC in this Linux environment by
using the guidelines that are laid out in this document.
The SVC is a storage-virtualization system that enables a single point of control for disparate,
heterogeneous storage resources to help support improved business application availability and greater
resource utilization. SVC is designed to pool storage volumes from IBM and other storage-system vendors
into a single reservoir of capacity for centralized management. SVC is designed to provide the following
benefits:
• Simplify storage management
• Reduce IT data storage complexity and costs while enhancing scalability
• Improve flexibility and responsiveness of the IT infrastructure
• Increase application availability by enabling changes to the storage infrastructure without impact to
servers

SVC offers network-based replication services that operate across multiple storage systems from different
vendors, which can help simplify the storage environment and reduce the total cost of storage. SVC
replication services include IBM FlashCopy®, IBM Metro Mirror and IBM Global Mirror.
A Device Mapper Multipath (DM) is used with SVC in multipath I/O (MPIO) environments that run on an IBM
System x™ server.
Oracle RAC 10g Release 2 and Oracle Clusterware technologies offer several features that work with both
single-instance database and clustered environments. Oracle ASM technology is a feature that is designed
to distribute information uniformly across all storage disk groups and has unique mirroring capabilities.

Deploying Oracle RAC 10g with ASM on Red Hat Linux and SUSE Linux with IBM SAN Volume Controller Ver: September 11, 2008
1

Introduction
Enterprise customers running Oracle Real Application Clusters (RAC) database can confidently use the IBM
System Storage SAN Volume Controller (SVC) to meet their critical business needs. SVC and Oracle RAC
database help to deliver robust, flexible, highly available, and cost-effective disk storage to support
continuous operations for mission-critical transactions.
This white paper explains the interoperability of SVC with Oracle RAC 10g Release 2 features, Oracle
Automatic Storage Management (ASM), and Oracle Clusterware on the Red Hat Enterprise Linux (RHEL) or
the SUSE Linux Enterprise Server (SLES) operating systems. Oracle ASM was introduced in Oracle
Database 10g and provides an integrated file system and volume manager for Oracle database files.
This white paper documents a configuration of Oracle RAC with ASM on a multinode cluster with the Linux
operating system and IBM System Storage SAN Volume Controller.

IBM System Storage SAN Volume Controller
The IBM System Storage SAN Volume Controller is designed to reduce both the complexity and costs of
managing your SAN-based storage. SAN Volume Controller helps you to achieve the following benefits:
• Simplify management and increase administrator productivity by consolidating storage management
from disparate disk systems into a single view

• Improve application availability by enabling nondisruptive data migration between disk systems.
• Improve disaster recovery and business continuance by applying and managing replication services
across disk systems within the SAN.
• Simplify device-driver configuration on hosts, so that all hosts within your network use the same IBM
device driver to access all supported storage systems through the SAN Volume Controller.
• Provide the following advanced features and functions to all servers that access SVC-managed
storage, such as:
• Large scalable cache

• Common replication services

• Centralized, standardized storage management

• Access to pooled storage that spans different disk systems

SVC nodes (also called engines) are the hardware elements of the SAN Volume Controller. SVC combines
pairs of nodes into a high-availability cluster. Each of the nodes in the cluster contains eight GB of high-
speed memory, which serves as the cache. The cluster is protected against data loss by uninterruptible
power supplies. The SVC nodes can only be installed in pairs for high availability.

SAN Volume Controller is not a RAID controller. The disk systems that are managed by SVC provide basic
RAID data protection. SVC builds virtual disks (Vdisks) by using managed disks that have been presented
by disk systems. Then, SVC presents the Vdisks to servers.

2

Replication services
SAN Volume Controller supports a broad range of replication services that operate in a consistent manner,
regardless of the type of storage that is being used with SVC. The FlashCopy function is designed to create
an almost instant copy of active data, which can be used for backup purposes or for parallel-processing
activities. Up to 16 copies of data can be created. Metro Mirror and Global Mirror operate between SVC
systems at different locations for business continuance purposes.
Management of the replication services is done through either the browser-based SAN Volume Controller’s
graphical user interface (GUI) or the command-line interface (CLI).

FlashCopy
The FlashCopy feature copies the contents of a source VDisk to a target VDisk. Any data that exists on the
target VDisk is lost and replaced by the copied data. After the copy operation completes, the target VDisks
contain the contents of the source VDisks as they existed at a point in time. Both source and target Vdisks
can then be updated independently. The FlashCopy feature is sometimes described as an example of a
point-in-time copy technology. Although the FlashCopy operation takes time to complete, the resulting data
on the target VDisk is presented so that the copy appears to occur immediately.

FlashCopy operations can occur on multiple source and target VDisks. FlashCopy management operations
are coordinated to allow a common single point in time for copying target VDisks from their respective
source VDisks. This allows a consistent copy of data that spans multiple VDisks. For SAN Volume Controller
version 4.2.0 or higher, the FlashCopy feature also makes it possible to copy multiple target VDisks from
each source VDisk. This can be used to create images from different points in time for each source VDisk.

Metro Mirror and Global Mirror
Metro Mirror and Global Mirror are continuous-replication functions. They operate between SVC systems at
different locations to help create copies of data for use in the event of a catastrophic data-center event.
Metro Mirror is designed to maintain a fully synchronized copy at metropolitan distances (up to 300 km)
whereas Global Mirror is designed to operate asynchronously and, thus, helps maintain a copy at much
greater distances (up to 8000 km). Because Global Mirror is an asynchronous function, the remote copy is
always lagging slightly behind the local copy and some updates are lost in the event of a disaster.

Note: For more details about copy services, see www.redbooks.ibm.com/abstracts/sg246423.html?Open.

Device Mapper Multipath
Device Mapper Multipath routes I/O over multiple paths that are accessible to the storage unit. If an active
path fails, Device Mapper Multipath reroutes the I/O over one of the other active paths.
Multipath awareness and support for an operating system can be described as being capable of doing the
following tasks:
• Provide a single-block device node for a multipathed LUN

• Ensure that I/O is rerouted to available paths when a loss of path occurs, with no userspace process
disruption other than a short pause
• Ensure that failed paths are revalidated as soon as possible

3

• Ensure stability of the naming of that node
• Configure the multipaths to maximize performance — spread I/O when possible path switching is
free, and not spread I/O when it is costly
• Configure the multipaths automatically and early (at boot) to permit the operating system to install
and boot on a multipathed LUN
• Reconfigure the multipaths automatically when events occur

Three spreading policies are currently implemented:
• failover: One path per priority group (PG). Therefore, I/O is routed to one path only.
• multibus: One PG containing all paths to the LUN. This brings the maximum spreading, but
assumes that all paths are excitable without penalty.
• group_by_serial: One PG per storage controller (serial); paths through one controller are assigned
to the associated PG. This policy applies to controllers that impose a latency penalty on the hand-
over of LUN management between a pair of redundant controllers.

These policies are stored in a config file (/etc/multipath.conf). Its content overrides the in-code defaults.

Oracle technologies
This section describes the Oracle RAC 10g R2 software and technologies.

Oracle RAC 10g Release 2 and Oracle Clusterware
Oracle 10g Release 2 provides features that can help ease complex Oracle administrative tasks. Some of its
key features are Oracle RAC, Oracle ASM, Oracle Clusterware (including Cluster Ready Services [CRS]),
enhanced Recovery Manager (RMAN) features, Flashback technology, performance-tuning enhancements,
job scheduler, Data Pump and Automatic Work Load Repository.

Oracle Cluster Ready Services (CRS)
Oracle 10g introduces Cluster Ready Services (CRS), a subcomponent of Oracle Clusterware. CRS
provides many system-management services to coordinate cluster-membership information, including the
following functions:
crsd: Performs high-availability recovery and management operations, such as maintaining the Oracle
Cluster Registry (OCR) and managing application resources. This process restarts automatically on failure.
evmd: Runs the event manager daemon.
ocssd: Manages cluster-node membership; failure of this process results in cluster restart.
oprocd: Process monitor for the cluster.
The CRS home is distinct from the Oracle RAC-enabled Oracle home. When vendor clusterware is present,
CRS interacts with it to coordinate cluster membership information.
The OCR contains cluster and database configuration information for Oracle Clusterware,including the list of
nodes in the cluster database, the CRS application, resource profiles, and the authorizations for the Event

4

Manager (EVM). The OCR can reside in a file on a cluster file system or on a shared raw device. When you
install Oracle RAC, you specify the location of the OCR.
CRS helps to package a set of application work under CRS control and provides access to the Oracle RAC
database. The application-resource profile defines the resources with which you manage the Oracle RAC.
In Oracle Database 10g, Cluster Synchronization Services (CSS) is the cluster manager on all platforms.
The Oracle Cluster Synchronization Service daemon (OCSSD) performs this function on platforms that are
based on the UNIX® operating system.

Oracle Cluster Synchronization Services
Automatic Storage Management (ASM) requires the use of Oracle Cluster Synchronization Services (CSS) to
enable synchronization between an ASM instance and the database instances that rely on it for file storage.
ASM is designed to work with both single database and Oracle RAC environments. In a single-instance
database environment, CSS maintains synchronization between the ASM and database instances. Oracle
Cluster Ready Services (CRS) includes the CSS component, which is automatically installed on each node
that runs ASM and is started when each node boots.
CSS monitors ASM and the shared-disk storage components, providing cluster and node-monitoring
management. Mounted ASM disk groups and ASM itself are registered with CSS upon startup, thereby
keeping disk-group metadata in synchronization across all Oracle RAC nodes. CSS thereby dynamically
registers any new ASM disk groups that are created and broadcast to the other cluster nodes.

Activities in both the database and ASM instances are synchronized using internode communication. CSS
verifies the health of those ASM instances. Structural changes that require synchronization, such as adding
or deleting a disk, initiate the internode messages. Efficient synchronization of both ASM and the database
are thereby accomplished with the same integrated lock-management infrastructure.

Establish device persistence with Oracle ASMLib
On Linux, various methods provide names that do not change, including devlabel and udev. What does
ASMLib provide that these solutions do not?
• Persistent naming-matching of a name with a set of permissions (make LUNs available to Oracle)
• Excluding names that match system disks to limit the disk ASMlib scans.

ASMLib never modifies the system's names for disks. ASMLib creates its own label for disk and provides its own
access to the devices with permissions for Oracle. The ASM disk names are placed in /dev/oracleasm/disks.
The configuration is persistent. ASMLib's labels are written to the ASM defined disks.

Oracle Automatic Storage Management (ASM)
As a vertically integrated file system and volume manager that was purposely built for Oracle database files,
ASM provides the performance of raw I/O with the easy management of a file system.
ASM provides a solution for storage-management challenges and an integrated storage-management
interface that maintains consistent volumes across servers and storage platforms. ASM virtualizes storage
into ASM disk groups. ASM distributes data evenly across storage resources within disk groups to optimize
performance and usage. ASM provides three mirroring options to protect against disk failure. (See Table 1.)

5

Mirror option Mirror description
External Defers redundancy to storage device
Normal 2-way mirroring
High 3-way mirroring
Table 1. Three mirroring options for protection against disk failure

ASM uses a unique mirroring algorithm. ASM does not mirror disks, but rather it mirrors extents. As a result,
a hot spare disk is not required. Only spare capacity is required within the disk group.
If a disk fails, ASM automatically reconstructs the contents of the failed disk on the surviving disks. Reading
the mirrored contents from the surviving disks, the I/O content that was hit from a disk failure is spread
across several disks, rather than on the single disk that mirrors the failed drive. When ASM allocates a
primary extent of a file to one disk in a disk group, it allocates a mirror copy of that extent to another disk in
the disk group. Primary extents on a given disk have their respective mirror extents on one of several partner
disks in the disk group. Each disk in a disk group has the same ratio of primary and mirror extents.

Advantages of ASM technology
There are many benefits to using ASM technology, including the following:
• Reduction of administration tasks
• Reduction in volume management tasks
• Automation of I/O tuning for workloads
• Simplification of the process for adding and removing disks
• Reduction in downtime as there is no file-system interface for file management
• Automatic balancing of load across disks in the ASM disk group — with the database up and running
after the changes to storage capacity

ASM disk groups
To reduce the complexity of managing ASM disk groups, Oracle recommends that generally no more than
two disk groups be maintained and managed per Oracle RAC cluster.
• Database Area: The active database files, control files, online redo logs, and change-tracking files
that are used in incremental backups are stored in this area
• Flash Recovery Area: Recovery related files, multiplexed copies of current control files and redo
logs, archive logs, backup sets, and flashback log files are stored in this area

ASM failure groups
Defining a failure group within ASM is an additional level of security against disk failure. A failure group is a
set of disks in a disk group that share a common resource. ASM ensures that a primary extent and its mirror
copy never reside in the same failure group. If you define failure groups for your disk group, ASM can
tolerate the simultaneous failure of multiple disks in a single failure group. This allows ASM to mirror across
disks on separate storage units to protect against the failure of an entire storage unit. ASM is able to take
advantage of database features to recover from I/O failures due to its tight integration with the database,.

6

Lab test environment
The following two tables provide the details related to the configuration and software levels for the servers
that are used for the example that is discussed in this white paper.

Oracle RAC servers
Table 2 shows the Oracle RAC servers running RHEL 4.4 that are used for the example presented in this
white paper.

Host – Server
Host type 4 x IBM System x3455
Processor 2 x dual-core AMD Opteron processor 2222 S 2992.634 Mhz
Memory 4 GB
HBA model QLE2462
Operating system Red Hat Enterprise Linux AS4 U4
Kernel version 2.6.9-42.ELsmp
Multipath software device-mapper-multipath-0.4.5-16.1.RHEL4
HBA driver 8.01.06
HBA firmware 4.00.23
Cluster file system None
Oracle software Oracle Database and Clusterware 10.2.0.3
Logical configure Four-node Oracle RAC cluster with ASM
Table 2. Oracle RAC servers

Table 3 shows the Oracle RAC servers running SLES 9 SP3 that are used for the example presented in this
white paper.

Host – Server
Host type 4 x IBM System x™ 3550
BIOS Version
Processor 2 x Dual-Core Intel® Xeon® processor 5160 @ 3.00 GHz
Memory 12 GB
HBA model Emulex 4G Model IBM 42C2071 2-Port PCIe FC HBA for System x
Operating system SUSE Linux Enterprise Server 9 SP3
Kernel version 2.6.5-7.244-smp
Multipath software device-mapper-1.0.1.01-1.6
HBA driver 8.0.16.27 HBAAP(I) v2.1.c, 02-02-06
HBA firmware 2.50A6 (Z2F2.50A6)
Cluster file system None
Oracle software Oracle Database and Oracle Clusterware 10.2.0.3
Logical configure Four-node Oracle RAC cluster ASM
Table 3. Oracle RAC servers

7

Storage hardware
Table 4 shows the storage hardware that is used for the example presented in this white paper.

Storage name OSL_SVC_11
Storage IP 9.11.113.35
Type SVC
Model 2145-8F4
Software level 4.2.1.1 (build 7.7.0711121400)
Table 4. Storage hardware

Topology of four-node Oracle RAC 10g Release 2 with ASM
Figure 1 shows the topology of the four-node Oracle RAC 10g Release 2 database with ASM

.
Figure 1. Four-node topology

8

Storage configuration by example
This section provides an example of the storage configuration scenario that is explained in this white paper.

Storage configuration on SVC
The SVC configuration defines the LUNs for presentation to the host nodes that will be used for the Oracle
ASM disk groups.

Understanding considerations for SVC configuration
When selecting managed disks (Mdisks) for managed disk groups (MDG), all LUNs for MDisk creation
need to have the same performance characteristics. If MDisks of different performance levels are placed
in the same MDG, the performance of the MDG can be reduced to the level of the poorest performing
MDisk. Likewise, all LUNs must also possess the same availability characteristics. Keep in mind that
SVC does not provide any RAID capabilities. Loss of access to any one of the MDisks within an MDG
impacts the entire MDG. Best practices for LUN selection include the following considerations:
• The LUNs must be the same type.
• The LUNs need to be the same RAID level.
• The LUNs must have the same RAID width (number of physical disks in the array).
• The LUNs must have the same availability and fault-tolerant characteristics.
• MDisks that are created on LUNs with varying performance and availability characteristics need
to be placed in separate MDGs.

Using SVC CLI to configure storage
Use the SAN Volume Controller CLI to set up the storage configuration:
SVC commands are divided into one of two categories: task-purposed (svctask) or informational
(svcinfo) commands. For a list of SVC commands, type in svctask –h or svcinfo –h and for a specific
command’s syntax, type in svctask <command> –h.

9

Creating virtualized storage from storage arrays
To create virtualized storage from storage arrays, perform the following steps:
1. List the unmanaged Mdisks that are available for adding to a managed disk group:
#>svcinfo lsmdisk –filtervalue mode=unmanaged
72 mdisk72 online unmanaged 50.0GB ……

2. Create an Mdisk group:
Syntax: svctask mkmdiskgrp -name mydiskgrpname –mdisk mdisk id_list -ext extent_size
#>svctask mkmdiskgrp -name mydiskgrp -mdisk 74:73:72, -ext 512

3. Change the Mdisk group name:
#>svctask chmdiskgrp -name ORA_Mdiskgrp mydiskgrp

4. View the Mdisk group:
#>svcinfo lsmdiskgrp
id name status mdisk_count vdisk_count capacity
8 ORA_Mdiskgrp online 3 0 150.0GB
extent_size free_capacity
512 150.0GB

5. View the Mdisk group in stanza format:
#>svcinfo lsmdiskgrp ORA_Mdiskgrp
id 8
name ORA_Mdiskgrp
status online
mdisk_count 3
vdisk_count 0
capacity 150.0GB
extent_size 512
free_capacity 150.0GB

10

Creating host connections
The prerequisites for the following steps are that the host has been zoned to SVC; that is, that the host
bus adapters (HBAs) are seen on the switch and are zoned to the SVC through host and SVC zoning:
1. List hba WWPNs:
#>svcinfo lshbaportcandidate
210100E08BBEE63A – qlogic hba
210000E08B9EE63A
10000000C93FA8FF – emulex hba

2. Create host identifier:
#>svctask mkhost -name ORA_NODE1 -hbawwpn 210100E08BBEE63A
add a second port to the same host for multipathing

#>svctask addhostport -hbawwpn 210000E08B9EE63A –force ORA_NODE1
note # (force parameter used for second wwpn on same host)

3. List host:
#>svcinfo lshost
id name port_count iogrp_count
8 ORA_NODE1 2 4

4. List host by ID to show WWPNs:
#>svcinfo lshost 8
id 8
name ORA_NODE1
port_count 2
type generic
mask 1111
iogrp_count 4
WWPN 210100E08BBEE63A
node_logged_in_count 2
state active
WWPN 210000E08B9EE63A
node_logged_in_count 2
state active

11

Creating virtual disks
Five 1 GB virtual disks are created for Oracle RAC: two OCR and three Vote disks.
25 GB Vdisks are created for the ASM disks.
1. Create the Vdisks from mdiskgroups
#>svctask mkvdisk -mdiskgrp ORA_Mdiskgrp -iogrp 0 -vtype striped -fmtdisk -
name OCR_1 -size 1 -unit gb

#>svctask mkvdisk -mdiskgrp ORA_Mdiskgrp -iogrp 0 -vtype striped -fmtdisk -
name ORA_ASM_1 -size 25 -unit gb

#>svcinfo lsvdisk

id name IO_group_id IO_group_name status mdgrp_id
37 ORA_ASM_1 0 io_grp0 online 5
mdgrp_name capacity type vdisk_UID
ORA_ASM_1 25.0GB striped 60050768019101D69000000000000037

#>svcinfo lsvdisk 37
id 37
name ORA_ASM_1
IO_group_id 0
IO_group_name io_grp0
status online
mdisk_grp_id 7
mdisk_grp_name ORA_Mdiskgrp
capacity 25.0GB
type striped
formatted yes
mdisk_id
mdisk_name
FC_id
FC_name
RC_id
RC_name
vdisk_UID 60050768019101D69000000000000037
throttling 0
preferred_node_id 25
fast_write_state not_empty
cache readwrite
udid

Tying the Vdisk to the host with mkvdiskhostmap
The following Linux mkvdiskhostmap commands tie the Vdisk to the host.
#>svctask mkvdiskhostmap -host ORA_NODE1 ORA_ASM_1
#>svctask mkvdiskhostmap -host ORA_NODE2 -force ORA_ASM_1
#note force parameter used to make vdisk available to second node

Note: For more information about the SVC CLI commands, visit the IBM Redbooks® Web site and
review the document entitled IBM System Storage SAN Volume Controller
(www.redbooks.ibm.com/redbooks/SG246423).

12

Configuration on server nodes
Configuration on the host requires defining the SVC devices in the multipath-configuration file.

Configuring DeviceMapper Multipath for mpio connections
Download an example of multipath.conf from www-
1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=ssg1S4000107&loc=en_US&cs=utf-
8&lang=en#DM.

Add the defaults, devices section and multipath entry for each LUN that is shared from the SVC to the
multipath.conf on the host.
Excerpt from the IBM multipath.conf file:
#defaults {
polling_interval 30
failback immediate
no_path_retry 5
rr_min_io 100
path_checker tur
user_friendly_names yes
}

devices {
# SVC
device {
vendor "IBM"
product "2145"
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_alua /dev/%n"

multipaths {
multipath {
wwid 3600507630efffe32000000000000120a
alias DATABASE_disk111
}
}

Example of a stanza for a disk that is set up for OCR on the SVC:
Note: Compare the last digits of vdisk_uid to the wwid in the entry:
VDISK_UID 60050768019101D69000000000000035
multipaths {
multipath {
wwid 360050768019101d69000000000000035
alias raw101
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_alua /dev/%n"
features "1 queue_if_no_path"
path_checker tur
path_selector "round-robin 0"
failback immediate
}
}

Note: For information about Device Mapper Multipath, visit: http://source.redhat.com/dm.

13

Oracle OCR and voting cluster disk raw devices example
The following listing shows an example of an Oracle OCR and voting cluster disk raw device:
vdisk id alias in multipath.conf
…30 raw101
…31 raw102
…32 raw103
…33 raw104
…34 raw105

Enable binding for the multipath devices to raw

For Red Hat Enterprise Linux:
/dev/raw/raw101 /dev/mapper/raw101

For SUSE Linux Enterprise Server:
raw101:mapper/raw101

Add permissions stanza to udev permissions

raw/raw101:root:dba:0640
raw/raw102:root:dba:0640
raw/raw103:oracle:dba:0660

Identifying storage on cluster nodes (SVC Vdisks for ASM)
ASMLib requires only one invocation of /etc/init.d/oracleasm scandisks to pick up all the changes
that were made on the other nodes.
• ASMLib and multipath disks

If multipathing is enabled, ASMLib must be configured to use the multipath devices. If the sd
devices are used, the system sees the same LUN through more than one of those sd devices.
ASMLib chooses the first path it found, negating the fault tolerance of multipathing.
• Multipath disks and ASMLIB configuration

Two important ASMLib configuration parameters in the oracleasm config file must be modified to
limit the scanning to only the multipath disks:
• ORACLEASM_SCANORDER=dm ensures that the device-mapper multipath devices are
scanned.
• ORACLEASM_SCANEXCLUDE=sd prevents the scanning of devices that Oracle cannot
use because of multipathing limits.

14

Best-practice recommendations
Here is a list of best-practices that you should follow:
• Use disks of similar characteristics within SVC when creating mdisk groups that support ASM-
assigned VDisks (LUNs).
• Use LUNs with similar capacity and performance within an ASM disk group.
• Use multiple LUNs per ASM disk group to evenly distribute database reads and writes across
members of the disk group.
• Place datafiles, control files, and redo logfiles in same ASM disk group. Multiplex control and redo
files to the flash recovery ASM disk group.
• Configure ASM disk groups to use external redundancy.
• Use Device Mapper Multipathing path-failover management.
• Use Vdisks with equal capacity and performance when adding space to an ASM disk group.

ASM simplifies database file management on System Storage disk products. External redundancy allows the
host to use more processor resources for data management and less processor resources for storage.
management. Oracle ASM and IBM SAN Volume Controller technologies work together to enhance
database administrator and system administrator productivity.

Summary
IBM System Storage SAN Volume Controller manages storage arrays to greatly simplify storage
administration for Oracle Database cluster running on ASM managed disks. SVC consolidates storage
subsystems into one logical storage system. ASM manages the workload of balancing I/O within the ASM
disk group, thus making the addition or removal of physical storage possible without having to shut down the
Oracle database. Although ASM is designed to provide fault tolerance through mirroring and striping, ASM
external redundancy takes advantage of existing storage fault tolerance mechanisms for increased data
protection.

15

Resources
These Web sites provide useful references to supplement the information contained in this document:
• IBM System p Information Center
http://publib.boulder.ibm.com/infocenter/pseries/index.jsp
• IBM Publications Center
www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi?CTY=US
• IBM Redbooks
www.redbooks.ibm.com

SAN Volume Controller
• IBM System Storage SAN Volume Controller
www.redbooks.ibm.com/abstracts/sg246423.html?Open
• The IBM System Storage SVC Information Center
http://publib.boulder.ibm.com/infocenter/svcic/v3r1m0/index.jsp
• IBM San Volume Controller Best Practices www.redbooks.ibm.com/redpieces/pdfs/sg247521.pdf

Device Mapper
• IBM support for Device Mapper
www-1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=
ssg1S4000107&loc=en_US&cs=utf-8&lang=en#DM

• For information about Device Mapper Multipath, visit: http://source.redhat.com/dm

Oracle
• Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g
Release 2 (10.2) Part No. B14201-04
http://download.oracle.com/docs/cd/B19306_01/install.102/b14203/toc.htm
• ASM 10gR2 New Features
www.oracle.com/technology/products/database/asm/pdf/asm%20r2%20new%20features.pdf
• ASM Best practices
www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bestpractices%2005-07.pdf
• Overview of Automatic Storage Management
www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf
• Configuring udev and device mapper for Oracle RAC 10g Release 2
www.expobadge.com/dldev/dc/DKLLoader1.cfm?dcid=153&type=pdf&aid=ace3f4dd-2baa-4ba9-
ad6c-099ce814bfba&caller=as1&shownumber=8027

16

Trademarks and special notices
© Copyright IBM Corporation 2008. All rights Reserved.
References in this document to IBM products or services do not imply that IBM intends to make them
available in every country.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked
terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was
published. Such trademarks may also be registered or common law trademarks in other countries. A current
list of IBM trademarks is available on the Web at "Copyright and trademark information" at
www.ibm.com/legal/copytrade.shtml.
UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.
All customer examples described are presented as illustrations of how those customers have used IBM
products and the results they may have achieved. Actual environmental costs and performance
characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published
announcement material, or other publicly available sources and does not constitute an endorsement of such
products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available
information, including vendor announcements and vendor worldwide homepages. IBM has not tested these
products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM
products. Questions on the capability of non-IBM products should be addressed to the supplier of those
products.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.

17

Deploying oracle rac 10g with asm on rhel and sles with svc

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to Deploying oracle rac 10g with asm on rhel and sles with svc (20)

Recently uploaded (20)

Deploying oracle rac 10g with asm on rhel and sles with svc