Computer Architecture and Organization

Computer Architecture and
Organization
Lecture Series for VIT, Chennai
Jaiber John, Intel
2022

Module 6
Computer Performance Analysis
Contents from:
1. [Stallings2016] Chapter 2
2. [Denis2020] Chapters 2, 4, 5

Computer Performance Analysis
● Performance Metrics and Evaluation
○ Historic Performance measures
○ Benchmarks
● CPU Performance bottlenecks
● Profiling
○ Vtune
○ Performance Optimization

Computer Performance
● Computer Performance depends
on:
○ CPU
○ Memory
○ IO
● Imbalance between growth of
performance
● Goal of System Architect is to:
○ Identifying bottlenecks
○ Compensate mismatch
○ Reduce power
consumption
○ Improve core performance
CPU vs. Memory vs. Disk speed trends

CPU Performance
● CPU Performance achieved by:
○ Increasing Clock speed
○ Increasing Logic density
○ Caching strategies
○ Exploiting parallelism (ILP, DLP, TLP)
○ Many Integrated Cores
○ Component Integration (GPU, IPU, various microcontrollers)
● How to evaluate performance across various CPU models in a
standardized way?

Moore’s Law and Dennard Scaling
● Moore’s Law (1965)
○ Number of transistors in a dense integrated circuit (IC) doubles about
every two years
○ Has slowed down since 2010
● Dennard Scaling (1974)
○ Known as MOSFET scaling, is a scaling law which states roughly that, as
transistors get smaller, their power density stays constant, so that the power
use stays in proportion with area; both voltage and current scale with length.
○ Ignored leakage current and threshold voltage
○ “Power wall” – occurs around 4 GHz

Amdahl’s Law (1967)
● The speedup using a parallel processor with N processors that fully exploits
the parallel portion of the program is :
Where, f -> fraction of time taken by parallel part of a program
(1 – f) -> fraction of time taken by sequential part of a program
N -> number of processors

Measures of CPU Performance
● Cycle time
f -> clock frequency
● Cycles per Instruction
-> no. of executed instructions of
type i
-> total number of execute
instructions
-> CPI of instruction type i
● Time taken to execute program:
● MIPS (Million Instructions per second)

Problem #1 (CPI, T, MIPS)
Given
• Total instructions (Ic) = 2 million = 2,000,000
• Frequency = 400 MHz = 400,000,000 Hz
Calculate avg. CPI, MIPS rate, program execution time T ?
CPI = 2.24 MIPS rate ~ 178 Program time T = 0.0139s

Arithmetic, Geometric, Harmonic Mean
For time-based
variables
For averaging
rates (MIPS,
MFLOPS)
For comparing
benchmarks
across systems

Performance Evaluation
● Theoretical approach – Performance Modeling and Simulation
○ While designing new systems
○ Queuing theory, discrete event simulation, Markov models, etc
○ Trace and event simulation, performance prediction
● Practical approach – Benchmarking
○ Run programs called “Benchmarks” on a computer and measure
various metrics
○ MIPS and MFLOPS cannot be directly compared across systems
○ SPEC, LINPACK, etc.. Popular benchmarks
○ VTune – collect and analyze “micro-architectural” events

Benchmarks
● SPEC
○ Standard Performance Evaluation Corporation (SPEC)
○ https://www.spec.org/benchmarks.html
● LINPACK/HPL (High Performance LINPACK) – For Supercomputer
performance
○ www.Top500.org
● MLBench – For Machine Learning performance

SPEC Flowchart
Normalized runtime ratio
Final metric (GM of ratios)

Performance Analysis using VTune
● Free Book: Performance Analysis and Tuning on Modern CPUs,
Denis Bakhvalov

Why Performance Engineering?
Speedups from performance engineering a program that multiplies two 4096-
by-4096 matrices

Performance Monitoring
● Software or Hardware Level
● Usages
○ Performance Tuning
○ Workload Characterization
○ Capacity Planning

Performance Evaluation with PMU
● PM - Critical step for
Performance Evaluation
● Different ways to improve
performance
○ Better Hardware
○ Better Algorithms
○ Optimizations in SW (Loop
vectorization, unrolling, etc)
○ Software Tuning
● Performance Monitoring Unit
(PMU)
○ Built-in processor core
PMU

Software Tuning by Performance Monitoring
● Frequent DRAM Access
● Cache Misses
● Page Faults
● Scheduling Overheads
● Core Underutilization
● Inefficient Synchronization
● .. Many more

Tools
● Intel ‘VTune Profiler’
○ Fix Performance Bottlenecks Quickly and Realize All the Value of
Hardware
● Intel ‘Advisor’
○ Efficient Vectorization, Threading, Memory Usage, and Accelerator
Offloading
● Linux perf

VTune – Full System Profiler
Analysis Group Analysis Types
Algorithm analysis Hotspots
Anomaly Detection
Memory Consumption
Microarchitecture analysis Microarchitecture Exploration
Memory Access
Parallelism analysis Threading
HPC Performance Characterization
I/O analysis Input and Output
Accelerators analysis GPU Offload
GPU Compute/Media Hotspots
CPU/FPGA Interaction
Platform Analyses System Overview
Platform Profiler

Software Tuning and Optimization with VTune
Develop and
build program
Profile app in
VTune
Review
hotspots and
anomalies
Fix, rebuild,
run program
Program
optimized
?
Stop
No Yes
Collect events
using ‘perf’
Analyse
events, identify
issues

Basic Analysis
● CPU Utilization
● Memory utilization
● CPI calculation
● Microarchitectural analysis
● Pipeline slots
● Cache Misses
● Branch mis-prediction

CPU Utilization
● Clock
○ Core clock – varies with frequency scaling
○ Ref clock – monotonic, counts independent of scaling
(TSC)
● If CPU Utilization low, app performance is low
● If CPU utilization too high, maybe it’s wasting CPU
cycles in busy loop

Cycles Per Instruction (CPI)
● Cycles Per Instruction (CPI) - how many cycles it took to retire one
instruction on
average.
● Instructions Per Cycle (IPC) - how many instructions were retired per
one cycle on
average.

VTune – Hotspot Analysis
● Hotspot – Identify portion of code where the program spends most of the time
● Focus on optimizing hotspots

VTune – Analysing Memory Access

VTune – Microarchitectural Exploration

Analysis of various Matrix multiply
functions and Optimization

Matrix Multiplication Sample code - VTune
● Summary: Multi-threaded sample code that implements several types of matrix
multiplication functions for analysis
● Placed under: [Documents]VTunesamplesmatrix
● Configurations:
○ NUM_THREADS or MAX_THREADS : 16
○ NUM or MSIZE : 2048
○ MATRIX_BLOCK_SIZE : 64
○ MULTIPLY : “multiply0”, “multiply1”, ... “multiply5”
● 3 Modes:
○ USE_THR - Use Threading
○ USE_OMP - Use OpenMP
○ USE_KML - Use Intel Kernel Math Library (Link)
● Runs on:
○ Windows and Linux

Program Flow
main() - matrix.c 64
ParallelMultiply() - thrmodel.c 79 (USE_THR), 171 (USE_OMP), 210 (USE_MKL)
ThreadFunction - thrmodel.c 49
MULTIPLY - multiply.c (any one of multiply0, multiply1, …)
Tutorial:
https://www.intel.com/content/www/us/en/develop/documentation/vtune-
tutorial-common-bottlenecks-windows/top.html

#1. Basic serial Implementation (USE_THR, multiply0)
void multiply0(int msize, int tidx, int numt, TYPE a[][NUM], TYPE b[][NUM],
TYPE c[][NUM], TYPE t[][NUM])
{
int i, j, k;
// Basic serial implementation
for(i=0; i<msize; i++) {
for(j=0; j<msize; j++) {
for(k=0; k<msize; k++) {
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}
} 1. Doesn’t use thread index, each loop multiples full matrix
2. Matrix is multiplied NUM_THREADS times
3. Very slow..

#2. Parallel multiply (multiply1, USE_THR)
void multiply1(int msize, int tidx, int numt, TYPE a[][NUM],
TYPE b[][NUM], TYPE c[][NUM], TYPE t[][NUM])
{
int i,j,k;
for(i=tidx; i<msize; i=i+numt) {
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}
}
1. Better than multiply0, since task is divided equally by threads
2. 1st thread processes 0th, 16th, 32nd .. element,
3. 2nd thread processes 1st, 17th, 33r, .. element, so on
4. Big performance problem: Accesses columns in inner loop

#2. Analysis of multiply1
● Sequentially accessing a column causes cache
thrashing and increases memory bandwidth
● Code is memory bound, has low IPC
(conversely high CPI rate)

#2. Analysis of multiply1
(Microarchitectural exploration)

#3. Loop interchange (multiply2, USE_THR)
void multiply2(int msize, int tidx, int numt, TYPE a[][NUM], TYPE b[][NUM], TYPE
c[][NUM], TYPE t[][NUM])
{
int i,j,k;
// This implementation interchanges the 'j' and 'k' loop iterations.
// The loop interchange technique removes the bottleneck caused by the
// inefficient memory access pattern in the 'multiply1' function.
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}
}
1. Loop interchange – row wise access
2. Avoids stride access, cache misses and
memory pressure
3. Improves performance

#3. Microarchitectural analysis of multiply2
1. DSB (Decoded Streaming Buffer) miss is high

#3. Turn on compiler optimization

#3. Microarchitectural analysis of multiply2
(After compiler optimization)

#4. With USE_OMP (OpenMP)
Change from USE_THR to USE_OMP in the Project Properties page

#4. With USE_OMP (OpenMP, multiply3)

Intel Advisor
Vectorization, Offload modeling, etc

VTune vs. Advisor
VTune
● Hotspot Analysis
● Threading Analysis
● Microarchitecture Exploration
● Memory Access Analysis
● IO Analysis
● GPU Analysis
Advisor
● Vectorization and Code
Insights
● CPU/GPU Roofline Insights
● Offload Modeling
● Threading

Heterogeneous Computing
● Parts of program executes across various compute units
○ CPU, GPU, FPGA, ASIC, TPU, etc
● Commonly used frameworks: CUDA, OpenCL, SYCL
● Programmer’s problems:
○ How to identify the heterogeneous compute units available
○ How to ascertain which parts of program to offload to compute units, to
achieve the best performance?
● Intel Advisor has ‘Offload Modeling’

Roofline Model Analysis
● Plots Machine Peak
performance against
Arithmetic intensity
● Arithmetic Intensity is ratio of
no. of operations per byte
read from memory
(FLOPS/byte)
● Helps in
○ Identifying machine
limitations
○ Optimization opportunities
in the program
○ Compare performance
across various architectures

Intel Advisor – Roofline Modeling
● Stride program
○ Scalar and vector loops
○ Row and column traversal
● Roofline Analysis
○ L1, L2, L3, DRAM Bandwidth
○ Loop location and analysis

Intel Advisor – Vectorization
● Before and after vectorization
○ Comparison of speed
○ Before: 27.48 s
○ GFLOPS 0.044
● Enable/Disable vectorization
○ Project Properties: /QxHOST /Ot

Intel Advisor – Roofline Model (Stride_and_MAP)
1. GFLOPS quite low
2. Low arithmetic intensity
3. 5 loops exhibit different Arithmetic intensity and GFLOPS

Enabling/Disabling Vectorization
To enable Vectorization:
- Optimization -> /O2
- Favor Size or Speed ->
/Ot
Refer to readme.txt for other
options

Analysis
● Analyse
○ CPI rate
○ Cache miss
○ Arithmetic Intensity
○ Branch prediction
○ Hotspots
● Scenario:
○ Stride
○ Vectorization
○ OpenMP enabling
○ Loop Unrolling effect

Case #1: Auto-Vectorization effects
void multiply3(int msize, int tidx, int numt, TYPE a[][NUM], TYPE
b[][NUM], TYPE c[][NUM], TYPE t[][NUM])
{
int i, j, k;
// Add compile option for vectorization report Windows: /Qvec-report3
Linux -vec-report3
#pragma ivdep
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}
}

#1. Analysis & Comments
Attributes No vectorization
(r013ue)
With auto-
vectorization (AVX2)
r0014ue
Comments
Execution time 12.33 s 2.894 s Perf. Boost is due to SIMD
CPI Rate 0.532 0.899 SIMD instructions can take more
cycles than scalar ones
Instructions Retired 318,820,000,000 41,188,200,000 SIMD instructions process more
data per instruction
CPU Front End usage 8.8 % 13.9 % No significant change
Bad Speculation 1.8 % 0.0 % No significant change
Pipeline slots utilization 87.3 % 52.5 % Indicates waiting for data
Memory bound 0.4 % 17.3 % SIMD needs more data bandwidth
Core bound 1.8 % 17.1 % SIMD lanes increases usage

Case #2: Row vs. Column access
// Before -> Column access
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}
// After –> Row access
c[i][j] = c[i][j] + a[i][k] * b[k][j];
}
}
}

Attributes Column Access
(r017ue)
Row access
(r0018ue)
Comments
Execution time 54.485 s 11. 676 s Accessing a column in a hot loop
causes lot of cache (cold) misses
CPI Rate 2.487 0.545 Clocks not wasted waiting for data
from memory
Pipeline slots utilization 19.9 % 90.0 % Better utilization, also indicates
vectorization could help here
Memory bound 64.4 % 0.2 % Significant reduction in memory
access (for row access) due to
caching
Core bound 11.1 % 0.8 % No significant change

Case #3. Loop Unrolling effects
for (i = tidx; i < msize; i = i + numt) {
for (k = 0; k < msize; k++) {
for (j = 0; j < msize; j+=8) {
c[i][j] = c[i][j] + a[i][k] * b[k][j];
c[i][j+1] = c[i][j+1] + a[i][k] * b[k][j+1];
c[i][j+2] = c[i][j+2] + a[i][k] * b[k][j+2];
c[i][j+3] = c[i][j+3] + a[i][k] * b[k][j+3];
c[i][j+4] = c[i][j+4] + a[i][k] * b[k][j+4];
c[i][j+5] = c[i][j+5] + a[i][k] * b[k][j+5];
c[i][j+6] = c[i][j+6] + a[i][k] * b[k][j+6];
c[i][j+7] = c[i][j+7] + a[i][k] * b[k][j+7];
}
}
}

Attributes Row access
(r0018ue)
Row access + loop
unrolling (r0019ue)
Comments
Execution time 11. 676 s 9.941 s Improvement in performance
due to loop unrolling
CPI Rate 0.545 0.498 Marginal change
Pipeline slots utilization 90.0 % 94.2 % Better utilization, also indicates
vectorization could help here
Memory bound 0.2 % 0.0 % No significant change
Branch instructions 17,318,789,664 896,020,160 ~19x reduction

Case #4. Branch misprediction (induced)
int direction[NUM];
// Randomize branch direction
for (i = 0; i < msize; i++) {
direction[i] = rand() % 2;
}
for (i = tidx; i < msize; i = i + numt) {
for (k = 0; k < msize; k++) {
for (j = 0; j < msize; j++) {
if (direction[j] == 1)
c[i][j] = c[i][j] + a[i][k] * b[k][j];
else
c[i][j] = c[i][j] + a[i][k] * b[k][j] + i;
}
}
}

Attributes Row access
(r0018ue)
Row access + branch
misprediction (r020ue)
Comments
Execution time 11. 676 s 27.90 s Less performance due to
speculation failure
CPI Rate 0.545 1.009 CPI drops due to pipeline stalls
Bad Speculation 0.4 % 37.8 % Big impact due to mis-predicted
branches
Pipeline slots utilization 90.0 % 49.1 % Pipeline underutilized due to
stalls
Memory bound 0.2 % 0.3 % No significant change
Branch instructions 17,318,789,664 29,069,454,048 ~2x increase

Case #5. Roofline impact of Vectorization on Loop
#define REPEAT 3000
#define SIZE 70000
for (int r = 0; r < REPEAT; r++)
{
for (int i = 1; i < SIZE; i++)
{
doublesArray[i] = i * 42.0 + doublesArray[i];
}
}
What are the loop dependences?

#5. Before Vectorization
doublesArray[i] = i * 42.0 + doublesArray[i];
Calculated:
Approximately 1 arithmetic ops for 16 bytes
So, arithmetic Intensity 1/16 = 0.0625
Observed:
Arithmetic intensity = 0.042 GFLOPS/byte
Performance = 0.685 GFLOPS
What would be the impact of vectorization?
Roofline model

#5. After Vectorization
Before:
● Arithmetic intensity = 0.042
GFLOPS/byte
● Performance = 0.685 GFLOPS
After:
GFLOP/byte
● Performance = 11.2 GFLOPs
Big improvement in performance, since
there is no loop carried dependence!

Case #6. Roofline impact of Vectorization on Loop
#define REPEAT 3000
#define SIZE 70000
for (int r = 0; r < REPEAT; r++)
{
for (int i = 1; i < SIZE; i++)
{
doublesArray[i] = i * 42.0 + doublesArray[i-1];
}
}
What are the loop dependences?

#6. Before Vectorization
doublesArray[i] = i * 42.0 + doublesArray[i-1];
Calculated:
Approximately 1 arithmetic ops for 16 bytes
So, arithmetic Intensity 1/16 = 0.0625
Observed:
Arithmetic intensity = 0.042 GFLOPS/byte
Performance = 0.689 GFLOPS
What would be the impact of vectorization?
Roofline model

#6. After Vectorization
Before:
GFLOPS/byte
● Performance = 0.689 GFLOPS
After:
GFLOP/byte
● Performance = 2.138 GFLOPs
Loop carried dependence has impacted the
performance gain from Vectorization

Computer Architecture and Organization

More Related Content

Similar to Computer Architecture and Organization (20)

Recently uploaded (20)

Computer Architecture and Organization

Editor's Notes