2020-ntn-vsphere_performance_principles_bondzio.pdf

DOAG 2020 │ ©2020 VMware, Inc.
ESXi Performance
Principles
DOAG Edition
Valentin Bondzio
Sr. Staff TSE / GSS Premier Services
2020-01-23

DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 2
Brief Intro

Brief Intro
@VMware since 2009
Global Support Services / Premier Services
Focus on Resource Management, Performance and Windows Internals
Originally from Berlin, living in Ireland since 2007
And most importantly …

Brief Intro
Not an Oracle expert !

DOAG 2020 NOON2NOON │ ©2020 VMware, Inc.
Agenda
6
CPU Scheduling and Usage Accounting
The “basics”
“Power Management”
The Good, the Better and the Ugly
ESXi Memory Management
More “basics”
Local resource distribution
What else is running on ESXi
CPU Topology Abstraction
CPU Socket != NUMA node

Agenda
7
The “basics”
More “basics”
+I/O stuff

Agenda
8
The “basics”
More “basics”
+I/O stuff
+vMotion

Agenda
9
The “basics”
More “basics”
+I/O stuff
+vMotion
+Backup

Resource guarantees and weighting (shares) on a per VM or “Resource Pool” level
CPU Scheduler Overview

Dispatch VMs (its “worlds”) to honor CPU settings (Local)
What does the scheduler do?
vCPU
HT / Core
vCPU
vCPU
vCPU vCPU vCPU

• For fairness: select VM with the least (consumed CPU time / fair share)
vCPU
HT / Core
vCPU
vCPU
vCPU vCPU vCPU

• For fairness: select VM with the least (consumed CPU time / fair share)
• For priority: run latency-sensitive VM (high) before anyone else
vCPU
HT / Core
vCPU vCPU
vCPU
vCPU vCPU vCPU
IO

LLC
Place the worlds / threads on physical CPUs (Global)
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
LLC

LLC
• To balance load across physical execution contexts (PCPUs)
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
LLC
VM VM VM VM

LLC
• To preserve cache state, minimize migration cost
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
LLC
VM VM VM VM

LLC
• To avoid contention from hardware (HT, LLC, etc.) and sibling vCPUs (from the same VM)
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
LLC
VM VM VM VM VM

LLC
• To avoid contention from hardware (HT, LLC, etc.) and sibling vCPUs (from the same VM)
• To keep VMs or threads that have frequent communications close to each other
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
Core
HT 0
HT 1
LLC
VM VM VM VM
VM VM
VM

How does that look?
10:10:29am up 2 days 48 min, 674 worlds, 1 VMs, 2 vCPUs; CPU load average: 0.02, 0.01, 0.01
PCPU USED(%): 0.3 0.1 0.0 0.3 0.2 0.1 0.0 0.0 0.0 0.2 50 50 4.1 0.1 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 AVG: 4.4
PCPU UTIL(%): 0.5 0.1 0.1 0.6 0.2 0.2 0.0 0.2 0.0 0.3 100 100 4.2 0.2 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.2 0.1 0.1 AVG: 8.6
CORE UTIL(%): 0.6 0.7 0.4 0.9 0.3 100 4.3 0.2 0.0 0.1 0.4 0.7 AVG: 9.1
ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP
96337 148153 vmx 1 0.02 0.01 0.02 61.82 - 37.86 0.00 0.00
96339 148153 NetWorld-VM-96338 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00
96340 148153 NUMASchedRemapEpochInitial 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00
96341 148153 vmast.96338 1 0.03 0.05 0.00 99.63 - 0.00 0.00 0.00
96343 148153 vmx-vthread-6 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00
96344 148153 vmx-mks:Debian86 1 0.00 0.00 0.00 61.55 - 38.13 0.00 0.00
96345 148153 vmx-svga:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00
96346 148153 vmx-vcpu-0:Debian86 1 62.35 99.68 0.00 0.00 0.00 0.00 0.00 0.05
96348 148153 vmx-vcpu-1:Debian86 1 62.36 99.67 0.00 0.00 0.00 0.01 0.00 0.05
96347 148153 PVSCSI-96338:0 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00
96350 148153 vmx-vthread-7:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00

?
CPU Usage Accounting
What states are there

Not Running
Running

Idle
(descheduled)
Running Ready

In an ideal world
Idle
(descheduled)
Running
Ready
Usage

What is charged against the VM
Idle
(descheduled)
Running
Ready
Usage Overlap HT busy Frequency ..

Idle
(descheduled)
Running
Ready
“stolen time”

Idle
(descheduled)
Running
Ready
“stolen time”
s
y
s
V
m
w
a
I
t
wait

Idle
(descheduled)
Running
Ready
“stolen time”
s
y
s
V
m
w
a
I
t
wait
C
S
T
P
R
D
Y
M
L
M
T

%LAT_C captures the gap between “ideal” execution (demand) and “current” execution.
• “Ideal”: unlimited dedicated cores running at nominal processor frequency
stolen time aka “%LAT_C”
Ideal Current
Demand

Ideal Current
%LAT_C
Demand

Ideal Current
%LAT_C
Sources of Compute Latency:
• VM resource contention: check %RDY and %CSTP
• Power management (P-State): frequency throttling
• Hardware contention: HTs are in use
Demand

Does enabling HT “spawn” a less capable “logical core”?
Intel® Hyper-Threading Technology
Cores and Threads

Cores and Threads
“physical” core
“logical”
core
“physical” core

Maybe two slightly less capable “logical” cores?
Cores and Threads
“physical” core
“logical”
core
“physical” core

Cores and Threads
“physical” core
“logical”
core
“physical” core
“physical” core
“logical”
core0
“logical”
core1

Individual throughput reduction, aggregated throughput increase at high load
100
100
~125

Intel® Hyper-Threading Technology on ESXi
Throughput reduction is accounted for in USED
100 100

100 100
125

100 100
125
2 x 50 + 12.5 = 62.5

100 100
125
HTEfficiencyShift – Default: 2
HT is:
1: 50 %
2: 25 %
3: 12.5 %
4: 6.25 %
5: 3.125 %
more efficient than no-HT
2 x 50 + 12.5 = 62.5

Usage vs. Utilization

Umbrella Term
Power Management

Umbrella Term
Power Management
P-States
Options aka: Power Regulator, CPU Power Management, EIST

Umbrella Term
Power Management
P-States
Deep C-States
Options aka: Power Regulator, CPU Power Management, EIST

Power Management refresher …
P-State = voltage / frequency point
C-State = idle state, running or varying degrees of stuff turned off
P2
P1
/ NF
P0
/ TB
Frequency
C0 C1-Cn
P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

C-State Transition

C1
C1
C1
C1
C-State Transition

C1
C1
C1
C1
C-State Transition
~1µs

Deep C-State Transition

C6
C6
C6
C6

C6
C6
C6
C6
~30µs

Dell
Power Management _Profiles_

ESXi Power Management Policy
Only affects what’s presented from the BIOS

Who controls what? → allow control /  use
CPU
BIOS
ESXi
VM /
guest

CPU
BIOS
ESXi
VM /
guest
deep C-
States
P-States
HLT / C1-Cn
P-States

CPU
BIOS
ESXi
VM /
guest
HLT / C1
deep C-
States
P-States
HLT / C1-Cn
P-States

Only affects what’s presented from the BIOS (DELL terminology)

System Profile → "Performance Per Watt (DAPC)"
"Performance Per Watt (OS)"
"Performance"
"Dense Configuration"
"Custom"

"Performance"
"Custom"
CPU Power Management → "System DPBM (DAPC)"
"OS DBPM"
"Maximum Performance“

"Performance"
"Custom"
"OS DBPM"
C States → "Enabled"
"Disabled"

"Performance"
"Custom"
"OS DBPM"
"Disabled"
P-States
P-States
P-States

"Performance"
"Custom"
"OS DBPM"
"Disabled"
P-States
P-States
P-States
C-States
C-States

Most likely …
Which BIOS policy am I running on?

Most likely “Dynamic”
Most likely …

Most likely …

Very likely “Performance”
Most likely …

4:30:58pm up 2 min, 1276 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.02, 0.00, 0.00
Power Usage: 94W, Power Cap: N/A
PSTATE MHZ:
CPU %USED %UTIL %C0 %C1 %C2 %A/MPERF
0 0.3 0.7 1 23 76 50.0
1 0.0 0.0 0 0 100 50.1
2 0.1 0.2 0 6 94 50.0
3 0.0 0.0 0 0 100 50.1
4 5.2 10.4 10 5 85 50.0
5 0.0 0.0 0 5 95 51.0
6 0.0 0.1 0 3 97 50.0
7 0.0 0.0 0 0 100 50.0
8 0.1 0.4 0 16 84 50.0
9 0.0 0.0 0 0 100 50.0
10 0.0 0.0 0 0 100 50.0
(…)

PSTATE MHZ:
CPU %USED %UTIL %C0 %C1 %C2 %A/MPERF
0 0.3 0.7 1 23 76 50.0
1 0.0 0.0 0 0 100 50.1
2 0.1 0.2 0 6 94 50.0
3 0.0 0.0 0 0 100 50.1
4 5.2 10.4 10 5 85 50.0
5 0.0 0.0 0 5 95 51.0
6 0.0 0.1 0 3 97 50.0
7 0.0 0.0 0 0 100 50.0
8 0.1 0.4 0 16 84 50.0
9 0.0 0.0 0 0 100 50.0
10 0.0 0.0 0 0 100 50.0
(…)

Most likely “Performance”
PSTATE MHZ:
CPU %USED %UTIL %C0 %C1 %A/MPERF
0 0.0 0.1 0 100 108.3
1 0.1 0.1 0 100 108.4
2 0.1 0.1 0 100 108.3
3 0.0 0.1 0 100 108.4
4 0.0 0.0 0 100 108.3
5 18.0 16.7 17 83 108.3
6 0.0 0.1 0 100 108.4
7 0.2 0.2 0 100 108.3
8 0.0 0.0 0 100 108.3
9 0.1 0.2 0 100 108.3
10 0.0 0.1 0 100 108.3
(…)

Most likely “Performance”
PSTATE MHZ:
0 0.0 0.1 0 100 108.3
1 0.1 0.1 0 100 108.4
2 0.1 0.1 0 100 108.3
3 0.0 0.1 0 100 108.4
4 0.0 0.0 0 100 108.3
5 18.0 16.7 17 83 108.3
6 0.0 0.1 0 100 108.4
7 0.2 0.2 0 100 108.3
8 0.0 0.0 0 100 108.3
9 0.1 0.2 0 100 108.3
10 0.0 0.1 0 100 108.3
(…)

Most likely “Custom”
PSTATE MHZ: 2401 2400 2300 2200 2100 2000 1900 1800 1700 1600 1500 1400 1300 1200
CPU %USED %UTIL %C0 %C1 %C2 %P0 %P1 %P2 %P3 %P4 %P5 %P6 %P7 %P8 %P9 %P10 %P11 %P12 %P13 %A/MPERF
0 0.2 0.4 0 16 83 62 0 0 0 0 0 0 0 0 0 0 0 0 38 75.2
1 0.0 0.0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 100 59.3
2 0.0 0.1 0 5 95 15 0 0 0 0 0 0 0 0 0 0 0 0 85 57.9
3 0.0 0.0 0 1 98 38 0 0 0 0 0 0 0 0 0 0 0 0 62 61.5
4 0.0 0.0 0 4 96 5 0 0 0 0 0 0 0 0 0 0 0 0 95 52.0
5 0.0 0.0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 100 50.3
6 0.1 0.1 0 1 99 7 0 0 0 0 0 0 0 0 0 0 0 0 93 67.7
7 0.1 0.1 0 0 100 99 0 0 0 0 0 0 0 0 0 0 0 0 1 77.7
8 0.0 0.0 0 0 100 10 0 0 0 0 0 0 0 0 0 0 0 0 90 50.8
9 0.0 0.1 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 100 51.6
10 0.0 0.0 0 3 97 8 0 0 0 0 0 0 0 0 0 0 0 0 92 54.0
(…)

The magic of Turbo Boost
Dynamic, supported overclocking
P1
TB1
Frequency
C0
C-State
depth
P1
TB1
C1 C1
C1

P1
TB1
Frequency
C0
C-State
depth
C6
P1
TB1
C1 C1
C1
P1
TB1
C0
P1
TB1
C6 C6
TB2 TB2
TB3 TB3
TB4 TB4

P1
TB1
Frequency
C0
C-State
depth
C6
P1
TB1
C1 C1
C1
P1
TB1
C0
C6 C6
TB2
TB3
TB4
TB5
C6
TB6
TB7

Power Policy “playfield"
BIOS “Dynamic” pre Haswell
Bad
Good
Optimal*

Bad
Good
Optimal*
BIOS “Dynamic” on Haswell+

BIOS “Maximum / High Performance”
Same* as Custom BIOS + High Performance ESXi policy (with the exception of C1E)
Bad
Good
Optimal*

BIOS “Maximum / High Performance”
Same* as Custom BIOS + High Performance ESXi policy (with the exception of C1E)
Custom BIOS + Custom or Balanced ESXi policy
Bad
Good
Optimal*
* a few workloads fare better with more deterministic performance

Custom done right!

Custom done right!
Custom BIOS
+
ESXi Balanced
“Dynamic”

Custom done right!
“Performance”
Custom BIOS
+
ESXi Balanced
“Dynamic”
Custom BIOS
+
ESXi Balanced
“Dynamic”

“Why doesn’t the frequency I see in Task Manager
change?”
Frequently Asked Questions
Power Management Trivia

change?”
• Possibility 1: You are looking at the brand string
• Possibility 2: You are looking in the right place
(but the guest OS has no way of knowing)

change?”
• Base frequency should be:
CPUID.(EAX=16h):EAX[15-00]
– But it seems Windows is getting that from SMBIOS

change?”
• Base frequency should be:
CPUID.(EAX=16h):EAX[15-00]
– But it seems Windows is getting that from SMBIOS
# grep cpuid ./WinTest.vmx
cpuid.16.eax = "----------------0100011100011000"
cpuid.coresPerSocket = "6"
cpuid.brandstring = "VMware (R) SuperSecretCPU (R) @ 18.2 GHz"

“I turned off all C-States, why is it still showing C1 in esxtop?”

PSTATE MHZ:
0 0.0 0.1 0 100 108.3
1 0.1 0.1 0 100 108.4
2 0.1 0.1 0 100 108.3
3 0.0 0.1 0 100 108.4
4 0.0 0.0 0 100 108.3
5 18.0 16.7 17 83 108.3
6 0.0 0.1 0 100 108.4
7 0.2 0.2 0 100 108.3
8 0.0 0.0 0 100 108.3
(…)

• You can’t turn off C1, you can disable different levels of deep C-States (C2+)
PSTATE MHZ:
0 0.0 0.1 0 100 108.3
1 0.1 0.1 0 100 108.4
2 0.1 0.1 0 100 108.3
3 0.0 0.1 0 100 108.4
4 0.0 0.0 0 100 108.3
5 18.0 16.7 17 83 108.3
6 0.0 0.1 0 100 108.4
7 0.2 0.2 0 100 108.3
8 0.0 0.0 0 100 108.3
(…)

“I won’t have any issues if I have everything set to High Performance in the BIOS, right?”

• No, besides possibly:
– PSU redundancy issues
– Power capping
– Temperature
– Firmware bugs

– Power capping
– Temperature
– Firmware bugs
• And definitely …
– No ability to control P-/deep C-States
– No maximum Turbo Boost frequencies …

– Power capping
– Temperature
– Firmware bugs
http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v3-spec-update.pdf

“I can clearly see C2 in perfmon on Windows,
why are you lying to me?”

“I can clearly see C2 in perfmon on Windows,
why are you lying to me?”
• This is either a perfmon bug or a choice to
represent
an “enlightened” idle feature
– “Intelligent Timer Tick Distribution (ITTD)”
– needs Windows 2012 R2 / vHW 11
– disable via “monitor.disable_guest_idle_msr = true”
• you really shouldn’t have to ever …

What runs where and when
The high level picture
CPU
VMK VMM
OS / APPs

Mostly Direct Exec
CPU
OS / APPs

Mostly Direct Exec
PCPU
vCPU
(…)
0xffffffff810a99d0 <+416>: test %eax,%eax
0xffffffff810a99d2 <+418>: je 0xffffffff810a9932 <cpu_startup_entry+258>
0xffffffff810a99d8 <+424>: callq 0xffffffff810c6ed0 <rcu_irq_enter>
0xffffffff810a99dd <+429>: mov 0x82740c(%rip),%r13
0xffffffff810a99e4 <+436>: test %r13,%r13
0xffffffff810a99e7 <+439>: je 0xffffffff810a9a07 <cpu_startup_entry+471>
0xffffffff810a99e9 <+441>: mov 0x0(%r13),%rax
0xffffffff810a99ed <+445>: no0xffffffff810a99f0 <+448>: mov 0x8(%r13),%rdi
0xffffffff810a99f4 <+452>: add $0x10,%r13
0xffffffff810a99f8 <+456>: xor %esi,%esi
0xffffffff810a99fa <+458>: mov %ebp,%edx
0xffffffff810a99fc <+460>: callq *%rax
0xffffffff810a99fe <+462>: mov 0x0(%r13),%rax
0xffffffff810a9a02 <+466>: test %rax,%rax
0xffffffff810a9a05 <+469>: jne 0xffffffff810a99f0 <cpu_startup_entry+448>
0xffffffff810a9a07 <+471>: callq 0xffffffff810c6e40 <rcu_irq_exit>
0xffffffff810a9a0c <+476>: jmpq 0xffffffff810a9932 <cpu_startup_entry+258>
0xffffffff810a9a11 <+481>: nopl 0x0(%rax)
0xffffffff810a9a18 <+488>: mov %gs:0xa0e4,%eax
0xffffffff810a9a20 <+496>: mov %eax,%eax
0xffffffff810a9a22 <+498>: bt %rax,(%rbx)
(…)

What about Idle?
CPU
vCPU
(…)
0xffffffff81052c20 <+0>: sti
0xffffffff81052c21 <+1>: hlt
*loud screeching sound*

VMM traps on the privileged instruction and puts (with VMK) the vCPU to “sleep
CPU
VMM
(…)
0xffffffff81052c20 <+0>: sti
0xffffffff81052c21 <+1>: hlt
*tells VMK to deschedule*

The scheduler decides what next to run
CPU
VMK

E.g. a vCPU / world that is ready to run
CPU
other vCPU

ESXi’s _own_ idle thread
CPU
C1-Cn

Manage host physical memory to abstract physical memory away from guest.
Allow memory over-commitment to provide an illusion of virtual DRAM to the guest.
Hide transient host memory pressure from application
Memory Management Overview
Goals and Objectives
Host Physical Memory Guest Memory
ESXi

Virtual Memory
Process 0

Virtual Memory
Process 0
Process 1
Process 2
Process 3
Process n

Virtual Memory
From the process’ point of
view, it provides:
• Contiguous address space
• Isolation / Security
Process 0
Process 1
Process 2
Process 3
Process n
256 TB
256 TB
256 TB
256 TB
256 TB

Virtual Memory
view, it provides:
Virtual Memory abstracts
Process 0
Process 1
Process 2
Process 3
Process n
Magic
256 TB
256 TB
256 TB
256 TB
256 TB

Virtual Memory
view, it provides:
• It provides the possibility to
overcommit …
Process 0
Process 1
Process 2
Process 3
Process n
Magic
256 TB
256 TB
256 TB
256 TB
256 TB

Virtual Memory
view, it provides:
overcommit …
The process is unaware what
is backing the virtual address
• Physical Memory
• Swap File
Process 0
Process 1
Process 2
Process 3
Process n
Magic
256 TB
256 TB
256 TB
256 TB
256 TB
64 TB
256 TB

Virtual Physical Memory
VM 0
Abstraction …

VM 0
VM 1
VM 2
VM 3
VM n
Abstraction …

From the VMs point of view,
it provides:
VM 0
VM 1
VM 2
VM 3
VM n
6 TB
6 TB
6 TB
6 TB
6 TB
Abstraction …

it provides:
Virt. Physical Mem. abstracts
VM 0
VM 1
VM 2
VM 3
VM n
Magic
6 TB
6 TB
6 TB
6 TB
6 TB
Abstraction …

it provides:
overcommit …
VM 0
VM 1
VM 2
VM 3
VM n
Magic
6 TB
6 TB
6 TB
6 TB
6 TB
Abstraction …

it provides:
overcommit …
The VM is unaware what is
backing the physical address
• Physical Memory
• Swap File
VM 0
VM 1
VM 2
VM 3
VM n
Magic
6 TB
6 TB
6 TB
6 TB
6 TB
16 TB
*** TB
Abstraction …

it provides:
overcommit …
• Physical Memory
• Swap File
• Or COW, ZIP, BLN
VM 0
VM 1
VM 2
VM 3
VM n
Magic
6 TB
6 TB
6 TB
6 TB
6 TB
16 TB
*** TB
*** TB
Abstraction …
*** TB

it provides:
overcommit …
• Physical Memory
• Swap File
• Or COW, ZIP, BLN
VM 0
VM 1
VM 2
VM 3
VM n
Magic
6 TB
6 TB
6 TB
6 TB
6 TB
16 TB
*** TB
*** TB
Abstraction …
*** TB
*** TB
*

Understanding VM memory usage on ESXi
How to Hide Memory Pressure?

Total Memory Size

Total Memory Size
Allocated Memory
Free Memory

Total Memory Size
Allocated Memory
Free Memory
Active Memory
Idle Memory

Reclaim memory from VM if it using more than it is entitled.
• Entitlement depends on configuration (reservation / shares / limit).
• Techniques to reclaim memory from VMs includes:
– Page sharing > Ballooning > Compression > Host swapping
– Breaks host large pages
Total Memory Size
Allocated Memory
Free Memory
Active Memory
Idle Memory

Active Memory
Not the same as guest stats!

Active Memory

Active Memory
!=

Active Memory
ESXi VM level heuristic
• Weighted, moving average
• OS / VMTools independent
• “Memory Sampling”
aka Touched

Active Memory
Un-maps 100 random pages
over the entire VMs mapped
address space
aka Touched
VM mapped memory
4 KB
100 x
4 KB 4 KB 4 KB 4 KB 4 KB 4 KB …

Active Memory
address space
Monitors R/W for a minute
(access traps to the VMM)
aka Touched
VM mapped memory
4 KB
100 x
4 KB 4 KB 4 KB 4 KB 4 KB 4 KB …
/ min

Active Memory
address space
aka Touched
VM mapped memory
4 KB
100 x
4 KB 4 KB 4 KB 4 KB 4 KB 4 KB …
/ min
Read
Read Write

Active Memory
address space
After one minute, re-maps all
remaining pages, starts again
aka Touched
VM mapped memory
4 KB
100 x
4 KB 4 KB 4 KB 4 KB 4 KB 4 KB …
/ min

Active Memory
vs. Consumed

Active Memory
What to trust?
consumed
active

Active Memory
What to trust?
active consumed

Guest Memory Metrics
In a nutshell

In a nutshell

Active Memory
Guests working set tends to be between active and consumed
consumed
active guest WS

Active Memory
Guest WS might over report (greedy app)
active guest WS

Active Memory
But guest WS will not underreport
consumed
active
guest WS

Active Memory
Not then end all of guest workload estimation

Hierarchical Resource Groups
From an ESXi perspective
host The host owns all resources

host
system vim iofilters user
The host owns all resources
Those are distributed by
hierarchical resource groups

host
minfree kernel helper ft drivers vmotion …

host
vmkboot CpuSched Init …

host
Consumers can demand
(request) resources
vmkboot CpuSched Init …

host
vCenter shows the sum of all
user resources as:
Total Reservation Capacity

host
user resources as:
Global Resource Pools are
then distributed back to
hosts into Local RPs
• Based on VMs demand
…
pool4
pool3
pool2
pool1

host
user resources as:
Global Resource Pools are
then distributed back to
hosts into Local RPs
…
vm.vmid
vm.vmid
vm.vmid
…
pool4
pool3
pool2
pool1

user Local Resource Groups are
created and incrementally
numbered when clients are
instantiated:
• VM starts / vMotions etc.
…
vm.vmid
vm.vmid
…
pool430
pool231
pool15
pool1

instantiated:
The local hierarchy is equal
to the global one
• Check for VM / LRG siblings
…
vm.vmid
vm.vmid
…
pool430
pool231
pool15
pool1
vm.vmid pool321
vm.vmid vm.vmid …

instantiated:
The local hierarchy is equal
to the global one
• Check for VM / LRG siblings
VM groups have multiple leaf
consumers
• vmid is local, not global
…
vm.vmid
vm.vmid
…
pool430
pool231
pool15
pool1
vm.vmid pool321
vm.vmid vm.vmid …
vmm uw ...

cpu.resv Reservation
cpu.limit Limit
cpu.shares Shares
cpu.resvLimit Expandable*
mem.resv Reservation
mem.limit Limit
mem.shares Shares
mem.resvLimit Expandable*
Memory
CPU
Both Memory and CPU resources
host
…
vm.vmid
vm.vmid
vm.vmid
…
pool4
pool3
pool2
pool1

ESXi CLI (via SSH)
… for CPU … for Memory … for comparison
Tools
sched-stats memstats esxtop

Tools
cmdline for local groups (no VMs)
sched-stats

Tools
# sched-stats -t groups | awk 'NR == 1
sched-stats

Tools
|| $2 ~ /^(vm.|pool)[0-9]+/
sched-stats

Tools
|| $2 ~ /^(vm.|pool)[0-9]+/
|| /^ +[0-4] /
sched-stats

Tools
|| $2 ~ /^(vm.|pool)[0-9]+/
|| /^ +[0-4] /
{printf ("%-10s%-12s%-9s%-6s%-6s%-6s%-9s%-6s%-9s%-9s%-10sn"
,$1, $2, $3, $6, $8, $9, $10, $11, $12, $13, $14)}'
sched-stats

Tools
|| $2 ~ /^(vm.|pool)[0-9]+/
|| /^ +[0-4] /
,$1, $2, $3, $6, $8, $9, $10, $11, $12, $13, $14)}'
vmgid name pgid vsmps amin amax minLimit units ashares resvMHz availMHz
0 host 0 933 1600 1600 1600 pct 4096000 5232 33168
1 system 0 659 10 -1 -1 pct 500 288 33168
2 vim 0 271 4944 -1 -1 mhz 500 4344 33768
3 iofilters 0 3 0 -1 -1 pct 1000 0 33168
4 user 0 0 0 -1 -1 pct 9000 0 33168
sched-stats

Tools
|| $2 ~ /^(vm.|pool)[0-9]+/
|| /^ +[0-4] /
,$1, $2, $3, $6, $8, $9, $10, $11, $12, $13, $14)}'
0 host 0 933 1600 1600 1600 pct 4096000 5232 33168
1 system 0 659 10 -1 -1 pct 500 288 33168
2 vim 0 271 4944 -1 -1 mhz 500 4344 33768
3 iofilters 0 3 0 -1 -1 pct 1000 0 33168
4 user 0 0 0 -1 -1 pct 9000 0 33168
sched-stats

Tools
cmdline for local groups (with VMs)
# memstats -r group-stats
-g0 -l2
-s gid:name:min:max::conResv:availResv
-u mb
| sed -n '/^-+/,/.*n/p'
---------------------------------------------------------------------------------
gid name min max conResv availResv
---------------------------------------------------------------------------------
0 host 97823 97823 28917 68907
1 system 20024 -1 20008 68923
2 vim 0 -1 3378 68907
3 iofilters 0 -1 25 68907
4 user 0 -1 5490 68907
---------------------------------------------------------------------------------
memstats

Tools
-g0 -l2
-u mb
| sed -n '/^-+/,/.*n/p'
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
0 host 97823 97823 28917 68907
1 system 20024 -1 20008 68923
2 vim 0 -1 3378 68907
3 iofilters 0 -1 25 68907
4 user 0 -1 5490 68907
---------------------------------------------------------------------------------
memstats

(N)UMA
+ terminology

DIMMs
(N)UMA
+ terminology

DIMMs
Socket / Package
(N)UMA
+ terminology
0

DIMMs
Socket / Package
NUMA node
(N)UMA
+ terminology
0

DIMMs
Socket / Package
NUMA node
(N)UMA
+ terminology
0
1

DIMMs
Socket / Package
NUMA node
Socket != NUMA node
(N)UMA
+ terminology
0
2
1

DIMMs
Socket / Package
NUMA node
Socket != NUMA node
LLC / DIE
(N)UMA
+ terminology
0
2
1

DIMMs
Socket / Package
NUMA node
Socket != NUMA node
LLC / DIE
(CoD, SNC / Zen1/2)
(N)UMA
+ terminology
0
2
1

Importance of Memory Access Latency
Jim Gray’s Storage Latency Analogy (slightly adapted)

You want to calculate a + b and the operands are in:

your head

your head
=
register / 1 cycle

your head
=
register / 1 cycle
this room

your head
=
register / 1 cycle
this room
=
L1-L2 / 10 cycles

your head
=
register / 1 cycle
this room
=
L1-L2 / 10 cycles
this building

your head
=
register / 1 cycle
this room
=
L1-L2 / 10 cycles
this building
=
DRAM / 100 cycles

your head
=
register / 1 cycle
this room
=
L1-L2 / 10 cycles
this building
=
DRAM / 100 cycles
Finland + Algeria

your head
=
register / 1 cycle
this room
=
L1-L2 / 10 cycles
this building
=
DRAM / 100 cycles
Finland + Algeria
=
Disk / 10^6 cycles

Numbers based on Intel i7-3770 @ 3.4 GHz

access size cycles ns
L3 / Last Level Cache
core
0
core
1
core
2
core
3
L1 L1 L1 L1
L2 L2 L2 L2
IMC QPI
DRAM

L1 32 KB 4-5 1.5
core
0
core
1
core
2
core
3
L1 L1 L1 L1
L2 L2 L2 L2
IMC QPI
DRAM

L1 32 KB 4-5 1.5
L2 256 KB 12 4
core
0
core
1
core
2
core
3
L1 L1 L1 L1
L2 L2 L2 L2
IMC QPI
DRAM

L1 32 KB 4-5 1.5
L2 256 KB 12 4
L3 8 MB 30 10
core
0
core
1
core
2
core
3
L1 L1 L1 L1
L2 L2 L2 L2
IMC QPI
DRAM

L1 32 KB 4-5 1.5
L2 256 KB 12 4
L3 8 MB 30 10
DRAM GBs 30+ 66*
core
0
core
1
core
2
core
3
L1 L1 L1 L1
L2 L2 L2 L2
IMC QPI
DRAM

N(UMA)
All sockets share the FSB to
the Northbridge and hence
the bandwidth
• NB also known as “Memory
Controller Hub” or MCH
Uniform memory access
latency between every CPU
and every DIMM
Von Neumann Bottleneck
getting worse with faster
CPUs / more RAM
Pre-Opteron/Nehalem
1 2
NB
0 3

N(UMA)
the bandwidth
and every DIMM
CPUs / more RAM
Pre-Opteron/Nehalem
1 2
NB
0 3

0 1
3 2
NUMA
Every NUMA node has its
own Integrated Memory
Controller (IMC)
• Some AMD’s (Bulldozer and
newer) have two nodes per
socket / package
Remote access has to go
over the interconnect and
remote CPU’s IMC
• This adds additional latency
making local and remote
access Non-Uniform
Post-Opteron/Nehalem

0 1
3 2
NUMA
Controller (IMC)
socket / package
remote CPU’s IMC
access Non-Uniform

0 1
3 2
NUMA
2 QPI / IC
CPU
/ns
0 1 2 3
0 72 291 323 294
1 296 72 293 315
2 319 296 71 296
3 290 325 300 71
local adjacent “routed”

CPU
/ns
0 1 2 3
0 136 194 198 201
1 194 135 194 196
2 201 194 135 200
3 202 197 198 135
0 1
3 2
NUMA
3 QPI / IC
local adjacent

0 1
3 2
NUMA
Basic Migration Types
NUMA clients (vCPUs +
memory) are kept local to a
home node
Balance migrations re-assign
the home node, memory
follows vCPUs!
Locality migrations set home
node to where the most
memory resides

0 1
3 2
NUMA
home node
follows vCPUs!
memory resides

NUMA migration incurs significant cost.
• All pages need to be remapped, i.e. %localMemory initially drops to 0% and slowly recovers.
• Copying memory pages across NUMA boundaries cost memory bandwidth.
NUMA Scheduler Consideration
Local Contention vs Remote Access

NUMA migration incurs significant cost.
• All pages need to be remapped, i.e. %localMemory initially drops to 0% and slowly recovers.
• Copying memory pages across NUMA boundaries cost memory bandwidth.
NUMA Scheduler Consideration
Local Contention vs Remote Access
0
10
20
30
40
50
60
70
80
90
100
0
1
2
3
4
5
6
7
8
9
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
%Local-Mem
#Migrations
time (30sec)
Memory Locality & NUMA-migrations
(with NUMA Migration)
%local #migrations
0
20
40
60
80
100
120
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
%Local
#Migrations
time (30sec units)
Memory Locality & NUMA-migrations
(No NUMA Migration)
%local #migrations

We had good(ish) reasonsos
vNUMA auto-sizing history
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 4.0 ESX 4.1 ESXi 5.0 ESXi 5.1 ESXi 5.5 ESXi 6.0
ESX 3.5

(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
My starting data @ VMware
ESX 3.5

(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
cpuid.coresPerSocket
ESX 3.5

CPS in GUI & supported
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 3.5

Max vSMP 8
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 3.5

Max vSMP 8
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
cpuid.coresPerSocket vNUMA
ESX 3.5

Max vSMP 32
Max vSMP 8
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 3.5

numa.vcpu.min = 9
Max vSMP 32
Max vSMP 8
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 3.5

numa.vcpu.min = 9
Max vSMP 32
Max vSMP 8
(…) 2007 2008 2009 2010 2011 2012 2013 2014 (…)
ESX 3.5
cpuid.coresPerSocket → numa.vcpu.maxPerVirtualNode

VPD doesn’t affect ESXi sched.
PPD does define ESXi NUMA sched.
• AKA NUMA client
Doesn’t influence ESXi sched.
Might influence Guest / App sched.
CPU Topology
vNUMA Topology
Two level’s of abstraction
Virtual and Physical Proximity Domains
VPD
PPD
CPS

• AKA NUMA client
CPU Topology
vNUMA Topology
VPD
PPD
C
PPD
VPD
C C C C C

• AKA NUMA client
CPU Topology
vNUMA Topology
VPD
PPD
CPS
PPD

• AKA NUMA client
CPU Topology
vNUMA Topology
VPD
PPD
CPS
PPD
VPD

Running Compute Intensive Benchmark
Case Study: Project Pacific
https://blogs.vmware.com/performance/2019/10/how-does-project-pacific-deliver-8-better-
performance-than-bare-metal.html

43.5% local memory access
on native Linux

43.5% local memory access
on native Linux
99.2% local memory
access on Pacific Cluster

250
IO stuff

vSphere 6.0 achieves Line Rate throughput on a 40GigE NIC
Throughput ↑ from 20.5 to 35.5 Gbps
CPU Used ↓ from 36 to 13 % (per Gbps)
Herculean Network IO

By default, vSphere tunes for lower CPU usage by batching I/O operations
Virtual NIC coalescing - recap
Trading CPU Cycles for Lower Latency
Network

• By default, that is also the case for the RX and TX path on vNICs (here vmxnet3)
• When disabled:
– Every packet received interrupts immediately
– Every packet will be issued immediately
Network

• When disabled:
Network

• When disabled:
1
1
Network

• When disabled:
1 2 3 4 5 6 7 8 9 .. .. ..
1 2 3 4 5 6 7 8 9 .. .. ..
Network

Possible Latency Optimizations
Network latency optimization on the VM level
Network

Disable LRO (Large Receive Offload)
• Host wide: “Net.Vmxnet3SwLRO = false”
• Small packets are no longer concatenated into larger ones
Network

Disable (vNIC) coalescing
• VMX option: “ethernetX.coalescingScheme = disabled”
• Issue TX immediately and immediately interrupt on RX
Network

Disable (vNIC) coalescing
• VMX option: “ethernetX.coalescingScheme = disabled”
• Issue TX immediately and immediately interrupt on RX
Disable Dynamic queueing
• NetQueue feature, load balances and combines less used queues
• Disabling guarantees a single queue for the VM
Network

Network – Recommendations
Use vmxnet3 Guest Network Driver
Very efficient and required for maximum performance=
Evaluate Disabling Interrupt Coalescing
Default mechanism may induce small amounts of latency in favor of throughout
It’s a 10Gb+ World
1Gb saturation is real, more bandwidth required today, especially in light of vSAN, MonsterVM vMotion
Use Latency Sensitivity High ‘Cautiously’
While it can reduce latency and jitter in the 10us use case, it comes at a cost with core reservations, etc
Requires FULL CPU and MEM reservation – or it won’t work and won’t tell you

Herculean Storage IO
• More than 1 Million IOPs from 1 VM
Hypervisor: vSphere 5.1
Server: HP DL380 Gen8
CPU: 2 x Intel Xeon E5-2690, HT disabled
Memory: 256GB
HBAs: 5 x QLE2562
Storage: 2 x Violin Memory 6616 Flash Arrays
VM: Windows Server 2008 R2, 8 vCPUs and 48GB.
Iometer Config: 4K IO size w/ 16 workers
Reference: http://blogs.vmware.com/performance/2012/08/1millioniops-on-1vm.html

Bare-metal to virtual TPC-C* gap then and now(ish)
* Non-complaint,
fair-use
implementation of
the workload on
Oracle 12c. Not
comparable to
official results.

* Non-complaint,
fair-use
implementation of
the workload on
Oracle 12c. Not
comparable to
official results.
-
30
%

* Non-complaint,
fair-use
implementation of
the workload on
Oracle 12c. Not
comparable to
official results.
-
30
%
-
10%

Scaling out vs. up on the same host to amortize overhead
1416.37
0
200
400
600
800
1000
1200
1400
1600
Baremetal tpsE
Throughput
Score
TPC-E on native HP Proliant DL 385 G8
http://blogs.vmware.com/vsphere/2013/09/worlds-first-tpc-vms-benchmark-result.html
http://www.tpc.org/4064 / http://www.tpc.org/5201
470.31
468.11
457.55
0
200
400
600
800
1000
1200
1400
1600
Virtual tpsE of 3 VMs running TPC-VMS
Throughput
Score
TPC-VMS on virtualized HP Proliant DL 385 G8
VM3
VM2
VM1

Storage I/O latencies are higher in virtual
The Problem - with Database Logs

Usually not a noticeable problem for Data IO
• Long (5+ ms) latency on HDDs
• Random I/O, Many threads banging on the same spindle(s)
• Even some SSDs are ~1ms

Not OK for Redo Log access

Not OK for Redo Log access
• Short (<<1ms latency)
• Sequential I/O, Single-threaded, Write-Only
• Typically a write-back cache in the HBA or the array
• Check the Top 5 wait events in Oracle AWR or equivalent database health reports

The Solution - Trade CPU Cycles for Lower Latency

But when sensing low IOPS, vSphere stops batching and switches to low latency mode

But when sensing low IOPS, vSphere stops batching and switches to low latency mode
• For lowest latency, put the log device on a vSCSI adapter by itself
• Batching and coalescing is on a per-vSCSI bus, not device(!) basis
• Explicit tuning can prove more effective though

Explicit workaround on the issuing path:
• Default is Asynchronous request passing from vSCSI adapter to VMKernel
– But dynamically adjust for low IOPS case

• To explicitly force immediate initiation of I/O operation (sync)
– scsiNNN.reqCallThreshold = “1”

Explicit workaround on the completion path:
• Default is coalescing of Virtual Interrupts
– vSphere automatically suspends interrupt coalescing for low IOPS workloads

Explicit workaround on the completion path:
• Default is coalescing of Virtual Interrupts
– vSphere automatically suspends interrupt coalescing for low IOPS workloads
• Or explicitly disable Virtual Interrupt Coalescing
– For PVSCSI: scsiNNN.intrCoalescing = “False”
– For other vHBAs: scsiNNN.ic = “False”

VMFS on par or faster than RDM (approx. 1%)
Reference: http://www.vmware.com/techpapers/2017/sql-server-vsphere65-perf.html
Myth Revisited: RDM versus VMFS

Storage – Recommendations
Use Multiple vSCSI Adapters
Allows for more queues and I/O’s in flight
Use pvscsi vSCSI Adapter
More efficient I/O’s per cycle
Don’t Use RDM’s
Unless needed for shared disk clustering, no longer a performance advantage
VMware Snapshots Should Be ‘Temporary’
Despite constant performance improvements, snapshots should not live forever, Co-Stop, Syncronous
Leverage Your Storage OEM’s Integration Guide
They provide necessary guidance around items like multi-pathing

282
vMotion

vMotion Workflow
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

vMotion Workflow
Create VM on Destination
1
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

Copy Memory
vMotion Workflow
1
2
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

Quiesce VM on Source
Copy Memory
vMotion Workflow
1
2
3
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

Copy Memory
vMotion Workflow
1
2
3
Transfer Device State
4 vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

Copy Memory
vMotion Workflow
1
2
3
Resume VM on Destination
4
5
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host

Copy Memory
vMotion Workflow
1
2
3
4
5
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host
Execution
Switchover
Time of 1 sec

Copy Memory
vMotion Workflow
1
2
3
Power Off VM on Source
4
5
6
vMotion Network
Datastore
Source
ESXi Host
Destination
ESXi Host
Execution
Switchover
Time of 1 sec

Memory Copy
Source VM Memory
Destination VM Memory
Phase 0:
Copy the VM’s 40GB of memory, trace pages. As we send that memory, the VM dirties 10GB
Iterative Memory Pre-Copy

Memory Copy
Source VM Memory
Phase 0:

Memory Copy
Source VM Memory
Phase 0:
Phase 1:
Retransmit the dirtied 10GB. In the process, the VM dirties another 3GB

Memory Copy
Source VM Memory
Phase 0:
Phase 1:
Phase 2:
Send the 3GB. While that transfer is happening, the VM dirties 1GB

Memory Copy
Source VM Memory
Phase 0:
Phase 1:
Phase 2:
Send the 3GB. While that transfer is happening, the VM dirties 1GB
Phase 3:
Send the remaining 1GB

vMotion of Oracle RAC
It’s been working for a while …

297
Confidential │ ©2018 VMware, Inc.
pre 6.5*
Trace Cost
LP remap
Prealloced memory
RDTSC cost
(SDPS)
Common Issues for Monster VMs

299
Performance
Troubleshooting

How to troubleshoot any issue
No matter how complicated

1. Identify a related system or component
that your team is not responsible for

2. Hypothesize that the issue is with that component

3. Assign the issue to the responsible team

3. Assign the issue to the responsible team
4. When proven wrong, go to 1.

Tuning guide for a completely different system
Some advanced option found on a blog
Vaguely fitting KB
etc.
Perfectly valid methods to “troubleshoot” or “tune”
/s

The biggest enemy
"XY Problem"
1. I have problem X
1. I have problem Y
Y

The biggest enemy
"XY Problem"
1. I have problem X
1. I have problem Y
2. Help me solve problem Y
Y

The biggest enemy
"XY Problem"
1. I have problem X
1. I have problem Y
3. Hey! I still have a problem
Y
?

The biggest enemy
"XY Problem"
1. I have problem X
2. I think it is because of Y
3. I have problem Y
Y
?

The biggest enemy
"XY Problem"
1. I have problem X
3. I have problem Y
X
Y
?

The biggest enemy
"XY Problem"
1. I have problem X
3. I have problem Y
tl;dr
don’t jump to conclusions
X
Y
?
!

Where to use caution
Believing anybody

Believing anybody
“Trust, but verify.“*

Believing anybody
* From the Russian proverb:
"Доверяй, но проверяй"
{Doveryai, no proveryai}
“Trust, but verify.“*

Comparing hosts, past and present, etc.
!=

Don’t assume newer == better
!=

Don’t assume newer == better
Identify all differences
!=

Relying on Traffic Light Dashboards alone

All metrics green?

All metrics green?
→ All good then! (false negative)

All metrics green?
Some metrics red?

All metrics green?
Some metrics red?
→ Something must be broken! (false positive)

Working through a list of known issues
Very good to start with!
• Don’t spend more than half and hour

Working through a list of known issues
Very good to start with!
• Don’t spend more than half and hour
Can be from different perspectives
• Application
• Resources, e.g.:
– CPU contention
– Memory pressure
– Disk latency
– Etc.

Apply different methodologies as needed
e.g. directionally

e.g. directionally
Top → Down: drill down from the application / its metrics
• app specific / difficult to "profile" the whole path

e.g. directionally
Bottom → Up: investigate from the resource point of view
• easy to run into false positives / not all resources evenly covered

e.g. directionally
Bottom → Up: investigate from the resource point of view
• easy to run into false positives / not all resources evenly covered
Recommendation: Bottom Up Checklist first

What makes you think there is a performance issue
Has it ever performed well
What has changed since
Can it be quantified
What else is affected
What is the timing
Is it reproducible
etc.
Ask questions
Good ones, preferably

Take notes along the way
seriously

Take notes along the way
seriously
"Remember kids, the
only difference between
science and screwing
around is writing it
down."

Provide an exact timeline
Part of notetaking but often forgotten
2017-11-28
23:00 UTC
Upgrade
2017-11-29
07:00 UTC
Issue first
noticed
2017-11-29
> 23:59 UTC
Tried
everything
under the sun
and wrote
down nothing
2017-11-30
08:00
Called
GSS

Be accurate and universal
https://xkcd.com/1179/

334
SR examples
“The case of the unexplained …”

Initial SR description:
• Oracle DB on virtual 64bit W2K8 three times slower than physical
• on 32bit W2K8 and 32/64bit RHEL5, only 5% slower than physical
• benchmarked with production equivalent test script
Example 1 – Oracle DB performance
Tales from GSS

Initial SR description:
• Oracle DB on virtual 64bit W2K8 three times slower than physical
• on 32bit W2K8 and 32/64bit RHEL5, only 5% slower than physical
• benchmarked with production equivalent test script
Troubleshooting in support:
• checked logs for errors
• basics like power management, limits, etc
• research if similar issues have been reported
Tales from GSS

Tales from GSS

Reproducing in-house:
Tales from GSS

• the customer provided two pre-configured VMs
Tales from GSS

• during initial run, the 64bit VM performed worse by a factor of 3
Tales from GSS

• during initial run, the 64bit VM performed worse by a factor of 3
• automated benchmark start and result collection, dropped to 1.6 on avg.
Tales from GSS

Tales from GSS

Murphy's law strikes:
Tales from GSS

• Minor configuration issues (DB not starting, tnsnames changes)
• Initial booking for lab server ran out and it was re-imaged
• Redeploy to local box was delayed due to a network issue
• Automation scripts had to be recreated
• Flashback store ran full
Tales from GSS

• Minor configuration issues (DB not starting, tnsnames changes)
• Initial booking for lab server ran out and it was re-imaged
• Redeploy to local box was delayed due to a network issue
• Automation scripts had to be recreated
• Flashback store ran full
Our Oracle DBA configured both VMs with a default config ….
Tales from GSS

Tales from GSS

Tales from GSS
"The more updates or inserts
in a workload, the more
expensive it is to turn on
block checking“

The benchmark was an insert loop…
Tales from GSS
"The more updates or inserts
in a workload, the more
expensive it is to turn on
block checking“

In a Nutshell …

Configuration issue
In a Nutshell …

Configuration issue
No virtualization fault
In a Nutshell …

Configuration issue
No virtualization fault
~70 hours
In a Nutshell …

2020-ntn-vsphere_performance_principles_bondzio.pdf

More Related Content

Similar to 2020-ntn-vsphere_performance_principles_bondzio.pdf (20)

Recently uploaded (20)

2020-ntn-vsphere_performance_principles_bondzio.pdf