Cloud Consolidation with Oracle (RAC) - How much is too much?

5/14/2012

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cloud Consolidation with Oracle (RAC)
– How much is too much?
Markus.Michalewicz@oracle.com Nitin.Vengurlekar@oracle.com
Senior Principal Product Manager Technical Director, Oracle RAC
2 Copyright © 2012, OracleRAC, Oraclereserved.
Oracle and/or its affiliates. All rights America Development, Oracle America

1

5/14/2012

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features
or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Agenda
• Database Cloud Architectures
• General Considerations
• Database Consolidation
– Memory Management
– CPU(_COUNT) Management
• A word on Instance Caging
– Database Resource Management
– Summary Database Consolidation

• Oracle RAC-specific Considerations for Consolidation
– Per Server Limits
– Real Time (RT) Processes
– Cluster Limits

• Summary and Q&A


2

5/14/2012

Database Cloud Architectures
Common building blocks are shared server and storage pools
Infrastructure Cloud Database Cloud
Database Cloud
DW CRM ERP DW ERP CRM DW ERP CRM
DB

DB
DB

DB

DB

DB

DB
OS OS OS

Hypervisor Hypervisor OS OS OS OS

Server Database Schema


General Considerations


3

5/14/2012

Oracle (RAC) Database Consolidation
Registered vs. Running
• Registered databases and instances could Registered:
potentially start and run at the same time. • n DB instances
are defined to
run on a machine
• Oracle’s Quality of Service Management (potentially)
or scripts can be used to model policies
to run certain databases only at certain
times; e.g. geographic region over time vs.

• Assume the cluster is PST based:
Running:
• EMEA based DBs run 10:00pm – 8am PST • Registered databases
• APAC based DBs run 6:00pm – 3am PST and instances are
(concurrently) running
• USA based DBs run 8:00am – 6pm PST (active workload)


Database Consolidation
• Simplest of all consolidation cases: Database Cloud
• One database Instance per Server
DW ERP CRM

• When consolidating more than one
DB

database on a server, consider the
server capacity with any DB added.

OS OS
• More details:
Best Practices for Database Consolidation in Private Clouds
http://www.oracle.com/technetwork/database/focus-
areas/database-cloud/database-cons-best-practices-1561461.pdf

• Exadata Database Machine-specific: Schema
http://www.oracle.com/technetwork/database/features/
availability/exadata-consolidation-522500.pdf


4

5/14/2012

Consider Workload Characteristics during Capacity Planning
Existing Workload
Utilization

Peak
The smaller this
Average gap, the better.

Time

New Workloads
Workload A OR Workload B
Utilization

Utilization

Time Time


Existing Workload Resulting Workload
Utilization

Peak
The smaller this
gap, the better.
Average

Peak Poor match:
Utilization

Time
Gap increases
(antagonistic)

+ Average

Workload B
Time
Utilization

Time


5

5/14/2012

Existing Workload Resulting Workload
Utilization

Peak
The smaller this
gap, the better.
Average

Utilization
Time Good match:
Peak
Gap decreases
Average

+
(complimentary)

Workload A
Time
Utilization

Time


Consider Workload Characteristics during Downtime
Utilization Normal Operation
Utilization

Peak
HR DW ERP CRM
Average

DBA
Time DBA
DBB
OS OS OS OS


6

5/14/2012

Consider Workload Characteristics during Downtime
Utilization Normal Operation
Utilization

Peak
HR DW ERP CRM
Average

DBA
Time DBA
DBB
During Downtime
Peak OS OS OS OS
Utilization

Average

Time


The Benefits of Standardization
Easier deployment and better predictability
• Standardization of software and
hardware simplifies planning
• Standardized hardware means a
Nodes

predictable behavior should demand
increase and additional hardware needs
to be added (horizontal scaling approach)

New
• Using “application profiling” (template Application A

based deployment) based on current
system(s) and performance baselines Large
OLTP
Medium
allows for a predictable deployment of OLTP
Small
new applications on the same system OLTP

using existing profiles. Then deploy


7

5/14/2012



Starting block: One Database instance per server
• Components to consider:

• Memory

• CPU
DB

• I/O

• Processes

OS
• Network


8

5/14/2012

One instance per server as the basis
• An Oracle database by default assumes that there is
only one database instance running on the server:
• Instance parameters are based on this assumption

• Consolidation changes that premise

• Main resources used:

DB
• Memory
• CPU
• I/O

• Resources “regulated by default”:
• Memory OS
• SGA / PGA Targets
• CPU


Recommendation 1: Manage Memory carefully (and dynamically)
• Avoid memory starvation and swapping
as it has negative impact on the system.
• Do not oversubscribe memory resources

• Define memory settings carefully – rule of thumb:
• For the OS in general:
• Shared Memory identifiers and segments
DB

• Use Hugepages, if possible – details: 80%
• MOS notes 361323.1 and 401749.1

• For OLTP applications:
• SUM (sga_target + pga_aggregated_target)
<= 80% of physically available memory per DB server
OS 20%
• For DW / BI applications:
• SUM (sga_target + 3* pga_aggregated_target)


9

5/14/2012

Recommendation 2: Use CPU_COUNT to “cage instances”
• CPU usage should be regulated.
• The OS scheduler schedules CPU as
requested by each individual instance.
• The OS scheduler does not know about the

DBB
priority of the various instances on the server.

• Use CPU_COUNT or ideally Instance Caging

DBA
• Instance Caging is configured in just 2 steps:
• 1. Set “cpu_count” parameter
• Max. number of CPUs the instance can use at any time
• 2. Set “resource_manager_plan” parameter OS
• Enables CPU Resource Manager
• E.g. out-of-box plan “DEFAULT_PLAN”


Using CPU_COUNT as a “central knob”
• CPU_COUNT regulates CPU usage and Data Structures
dependent resources (to a certain degree): Concurrency

Parallelism
• Parallelism (PQ operations) fx(CPU_COUNT)
• Processes Processes
DBB

• Load Calculation Memory Allocation

Load Calculation

• Processes and PQ
DBA

operations should be 80%
considered explicitly.

16 OS 20%


10

5/14/2012

A Word on Instance Caging


Instance Caging: Partitioning Approach

• Provides maximum isolation
16 CPUs 32

28
• For performance-critical databases
DBA DBB DBC DBD

24

• If one database instance is idle, 20
its CPU allocation is unused 16
Instance D: 4 CPUs
12
• The rule of thumb for the partitioning Instance C: 4 CPUs
8
approach is to set a general limit: Instance B: 4 CPUs
• SUM (CPU_COUNT) < 75% x Total CPUs 16 OS 4
Instance A: 4 CPUs
0


11

5/14/2012

Instance Caging: Over-Provisioning Approach

• Best used for non-critical databases
16 CPUs 32
that are typically well-behaved
28

DBA DBB DBC DBD
• Contention for CPU if database 24
instances are sufficiently loaded 20
Instance D: 8 CPUs
16
• Typically not enough contention
12
to destabilize OS or DB instances Instance C: 6 CPUs
8
Instance B: 4 CPUs
• Best approach if the goal 16 OS 4
Instance A: 4 CPUs
is fully utilized CPUs 0


Instance Caging – under the covers
• If cpu_count is set to 4 on a 16 CPU server Over-Provisioning
• All foreground processes make progress Partitioning Approach Approach
• But only 4 foregrounds are running at any time 32 32
28 28
• Most backgrounds are not managed 24 24
• Critical and use very little CPU
20 20
• MMON, Job Scheduler slaves are managed Instance D: 8 CPUs
16 16
Instance D: 4 CPUs
• No CPU affinity 12 12
Instance C: 6 CPUs
Instance C: 4 CPUs
• Not meant for hard-partitioning or licensing 8 8
• All CPUs may be used Instance B: 4 CPUs Instance B: 4 CPUs
4 4
• CPU utilization averaged across all CPUs ≤ 25% Instance A: 4 CPUs Instance A: 4 CPUs
0 0
• More information:
http://www.oracle.com/technetwork/database/focus-areas/performance/instance-caging-wp-166854.pdf


12

5/14/2012

Over-Provisioning Approach – It’s still hardware that’s the limit
• Best used for non-critical 140
120
databases that are typically 100
80
well-behaved – examples:
CPU Util.
60
Average
40
• Complimentary workload 20
0
• Systems with little contention t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12

for CPU if database instances 32
16 CPUs
are sufficiently loaded. 28

DBADBBDBCDBD
24

• Do not use, 20
Instance D: 8 CPUs
16
• If the load is significant and
of longer duration, as system 12 Instance C: 6 CPUs
stability can get impacted. 8
Instance B: 4 CPUs
• For highly critical systems 16 OS 4
Instance A: 4 CPUs
0


How much over-provisioning is OK?
• As CPU_COUNT does not consider the
“quality of a CPU” an absolute maximum
32
is hard to determine / depends on the system. 16 CPUs
28
DBA DBB DBC DBD

• The general rule of thumb is:
24
• SUM (CPU_COUNT) <= up to 2x Total CPUs
20
• Consider different system types: Instance D: 8 CPUs
16
Threaded Core based Engineered
12
“Total CPUs” Instance C: 6 CPUs
# of threads # of cores # of threads
are based on 8
Max. over- Instance B: 4 CPUs
provisioning
1.5 (HT*)
2.0
3.0 (DBM) 16 OS 4
1.0 (hHT**) 2.0 (ODA) Instance A: 4 CPUs
factor
0
(HT*): ratio 1:2; (hHT**): ratio 1:n, with n >2


13

5/14/2012

Database Resource Management


Recommendation 3: Use DB Resource Manager
3 steps to use Resource Manager:

1. Group sessions with similar
performance objectives into
Consumer Groups

2. Allocate resources to consumer
groups using Resource Plans

3. Enable Resource Plan


14

5/14/2012

Summary Database Consolidation


Summary
1. Define memory settings carefully:
• For OLTP applications: 32
• SUM (sga_target + pga_aggregated_target) 28
DBB

• For DW / BI applications: 24

• SUM (sga_target + 3* pga_aggregated_target) 20
<= 80% of physically available memory per DB server 80% Instance D: 8 CPUs
DBA

16
2. Use CPU_COUNT or ideally Instance Caging 12
Instance C: 6 CPUs
• SUM (CPU_COUNT) <= up to 2 x Total CPUs 8
OS 20% Instance B: 4 CPUs
4
• Consider different system types. Instance A: 4 CPUs
0
3. These are the most crucial per server limits.


15

5/14/2012

Oracle RAC-specific
Considerations for Consolidation


Oracle RAC based Consolidation
Server limits are mostly reached before cluster limits apply
• Most customers will experience “per
server limits” before “cluster limits” apply.
• Oracle RAC introduces a few more
Per Server Limits

DBA1 DBB1 DBC1

DBA2 DBB2 DBC2

processes (potential limits) to consider.

• Oracle RAC DBs use LMS Real
Time (RT) processes per instance.
• LMS RT processes need to
be considered in particular
Clusterware Clusterware
16 OS 16 OS

Cluster Limits

16

5/14/2012

Real Time (RT) Processes


Considerations for Real Time (RT) Processes in general
• A Real Time process can only
run on one CPU (core) at a time.
• The usage of the CPU is typically short.
DBA1 DBB1 DBC1

DBA2 DBB2 DBC2

• The aggregated number of
RT processes per server should
not exceed the number of cores per server
• One Oracle RAC instance has typically at Clusterware Clusterware
least one RT process (LMS) per default
16 OS 16 OS
• An Oracle ASM instance has one RT process
• Oracle Clusterware uses various RT processes


17

5/14/2012

Background for LMS Real Time (RT) Process recommendation
• The number of LMS RT processes per
instance is determined by a function on
CPU_COUNT.
• In order to guarantee optimized

DBA1 DBB1 DBC1

DBA2 DBB2 DBC2
performance and reliability, the
general rule of thumb for RAC is:
• The aggregated number of LMS
RT processes per server should
not exceed [cores per server]-1
• See MOS note: 558185.1 for details
Clusterware Clusterware
• This leaves one core free for additional RT
processes to be assigned as needed, as 16 OS 16 OS
LMS RT can stay on a core for a moment


Automatic adjustment of LMS process priority in 11.2.0.3
• With 11.2.0.3 the number of LMS RT
processes are monitored and adjusted
according to the number of cores on
DBA2 DBB2 DBC2 DBD2
DBA1 DBB1 DBC1 DBD1

the node periodically.
Per Server Limits

DBF1 DBF2
• The goal is to keep
RT LMSs per server <= # cores per server
• For details, see MOS note 1392248.1 –
Auto-Adjustment of LMS Process Priority
in Oracle RAC with 11.2.0.3 and later DBE1 DBE2
• This excludes any ASM instance running
on the system as well as any pre-11.2.0.3
database instance 4 OS 4 OS
• See also MOS note 1439551.1 –
Oracle (RAC) Database Consolidation Guidelines
for Environments using mixed Database Versions


18

5/14/2012

Recommendation 4: Over-provision only based on CPU_COUNT
• Do not over-provision the number 48
.
of LMS RT processes on one server. .
.
.

DBA2 DBB2 DBC2 DBD2
DBA1 DBB1 DBC1 DBD1
.

Per Server Limits
.
• Limit the number of RT LMSs by: DBF1 DBF2

32
• Using CPU_COUNT
28
• Directly reducing the number of LMS 24
RT processes (gcs_server_processes)
DBE1 DBE2
20
• Downgrading additional LMS RT Instance D: 8 CPUs
16
processes to time share (TS).
12
• In 11.2.0.3 and later, the RT to CPU OS OS 8
Instance C: 6 CPUs

rule will be enforced by automatically Instance B: 4 CPUs
downgrading subsequently started 4
LMS RT processes to TS. Instance A: 4 CPUs
0


Cluster Limits


19

5/14/2012

Cluster Limits – Starting DBs are currently the main concern
• In most cases, per server limits will be Starting:
reached before cluster limits are reached. • Registered databases
and instances start
• Cluster limits apply to: • Default is “starting
• Registered resources (databases) at the same time”.

DBC1 DBD1

DBC2 DBD2
DBA1 DBB1

DBA2 DBB2
• Starting databases in the cluster – reason:
• Starting databases need to register
with the cluster (Oracle Clusterware).
• Currently and for example, 100 Clusterware Clusterware
starting Oracle RAC databases
on a 4 node cluster are supported. OS OS
• The Number assumes each Oracle RAC
database uses an instance on each node.
Cluster Limits

Summary


20

5/14/2012

(Oracle RAC) Database Consolidation
Summary
• For a successful database consolidation,
consider the following as rules of thumb: Existing Workload

Peak

DB
Utilization
80%
1. General considerations for capacity planning Average

2. Manage Memory carefully (and dynamically) Time

+ OS 20%
3. Use CPU_COUNT
4. Use DB Resource Manager
Data Structures
5. Over-provision only based on CPU_COUNT
Concurrency

Parallelism

Processes

Memory Allocation

Load Calculation
=


21

Cloud Consolidation with Oracle (RAC) - How much is too much?

More Related Content

What's hot (20)

Similar to Cloud Consolidation with Oracle (RAC) - How much is too much? (20)

More from Markus Michalewicz (20)

Recently uploaded (20)

Cloud Consolidation with Oracle (RAC) - How much is too much?