Performance Analysis with Scalasca on Summit Supercomputer part I

ORNL is managed by UT-Battelle, LLC for the US Department of Energy
Performance Analysis with Scalasca
George S. Markomanolis
7 August 2019

22 Open slide master to edit
Outline
• Introduction to Scalasca
• How to compile (using Score-P)
• Explaining functionalities of Scalasca/CUBE4 on two applications
• Testing a case with a large trace

Scalasca
• Scalasca is a software tool that supports the performance analysis of
parallel applications
• The analysis identifies potential performance bottlenecks – in particular
those concerning communication and synchronization – and offers
guidance in exploring their causes.
• Installed version on Summit: v2.25
• Module: scalasca
• For instrumentation is used Score-P
• Web site: https://www.scalasca.org
• Email: scalasca@fz-juelich.de

Capability Matrix - Scalasca
Capability Profiling Tracing Notes/Limitations
MPI, MPI-IO Yes Yes
OpenMP CPU Yes Yes
OpenMP GPU No No
OpenACC No No Score-P does instrument but CUBE does
not provide information
CUDA No No Score-P does instrument but CUBE does
not provide information
POSIX I/O Yes Yes
POSIX threads Yes Yes
Memory – app-level Yes Yes
Memory – func-level Yes Yes
Hotspot Detection Yes Yes
Variance Detection Yes Yes
Hardware Counters Yes Yes

Techniques
• Profile analysis:
– Summary of aggregated metrics
• Per function/call-path and/or per process/thread
– mpiP, TAU, PerfSuite, Vampir
• Time-line analysis
• Pattern analysis

Automatic trace analysis
• Apply tracing
• Automatic search for patterns on inefficient behavior
• Classificaiton of behavior
• Much faster than manual trace analysis
• Scalability

Workflow

CUBE4
• Parallel program analysis report exploration tools
• Libraries for XML report
• Algebra utilities for report processing
• GUI for interactive analysis exploration
• Three coupled tree browsers
• Performance property
• Call-tree path
• System location
• CUBE4 displays severities
• Value for precise comparison
• Colour for easy identification of hostpots
• Inclusive valye when closed and exclusive when expanded

Scalasca on Summit
module load scalasca
scalasca
Scalasca 2.5
Toolset for scalable performance analysis of large-scale parallel applications
usage: scalasca [OPTION]... ACTION <argument>...
1. prepare application objects and executable for measurement:
scalasca -instrument <compile-or-link-command> # skin (using scorep)
2. run application under control of measurement system:
scalasca -analyze <application-launch-command> # scan
3. interactively explore measurement analysis report:
scalasca -examine <experiment-archive|report> # square
Options:
-c, --show-config show configuration summary and exit
-h, --help show this help and exit
-n, --dry-run show actions without taking them
--quickref show quick reference guide and exit
--remap-specfile show path to remapper specification file and exit
-v, --verbose enable verbose commentary
-V, --version show version information and exit

Scalasca Workflow
• Compilation: use Score-P
• Execution of the binary for profiling (it will create an output folder):
scalasca -analyze jsrun …
• Examine of the data (GUI is loaded)
scalasca -examine output_folder

MiniWeather – MPI – Tools parameters
• Parameters for Scalasca/Score-P
module load scorep/6.0_r14595
module load scalasca/2.5
export SCOREP_METRIC_PAPI=PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_FP_OPS
export SCOREP_MPI_ENABLE_GROUPS=ALL
export SCAN_MPI_LAUNCHER=jsrun

Instrumentation

MiniWeather - MPI
• Edit the Makefile and add the $PREP before the compiler name
• Compile:
MPI: make PREP="scorep --mpp=mpi --pdt" mpi
• Execution (submission script):
scalasca -analyze jsrun -n 64 -r 8 ./miniWeather_mpi
• Visualize:
scalasca -examine /gpfs/…/scorep_miniWeather_mpi_8p64_sum

CUBE4 – Central View
3 Windows:
Metric tree
Call tree
System tree

Exploring the menus

How to expand the trees

Computation – System tre

Computation – Blox plot

Computation Sunburst

Computation – Process x Thread

Score-P configuration parameters

CUBE – Flat view

Initialization variation

MPI_Comm_dup variation

MPI_Comm_dup variation II

Getting information about metrics

MPI_Allreduce variation

Collective I/O

MPI_Waitall variation

Information about transferred data

Read-Individual operations

Write-Collective operations

Computational imbalance - Overload

Computational imbalance - Underload

Computational imbalace

Bytes read

Instructions

CUBE4 – Derived metrics – Instructions per Cycle
• Right click on any metric of the metric tree, and
select “Edit metric” -> Create derived metric ->
“as a root”

Instructions per cycle (IPC) – Useful computational
workload
There is no specific rule but codes with IPC less than 1.5, can be improved

Floating operations NOT per second

Derived metric of Floating operations to create the
metric Flops

Difference between two executions
cube_diff scorep_miniWeather_mpi_4p64_sum1/profile.cubex
scorep_miniWeather_mpi_8p64_sum2/profile.cubex -c -o result.cubex
cube result.cubex

Tracing with Scalasca
• Tracing can/will cause bigger overhead during the execution
of the application
• More information are recorded including timeline
• Scalasca will analyze the trace according to various patterns
and it will identify the bottlenecks

How much memory buffer to use for tracing?
• Examine the profiling data
scalasca -examine -s /gpfs/alpine/.../scorep_miniWeather_mpi_8p64_sum
INFO: Score report written to /gpfs/alpine/…/scorep_miniWeather_mpi_8p64_sum/scorep.score
• head /gpfs/alpine/…/scorep_miniWeather_mpi_8p64_sum/scorep.score
Estimated aggregate size of event trace: 978MB
Estimated requirements for largest trace buffer (max_buf): 16MB
Estimated memory requirements (SCOREP_TOTAL_MEMORY): 18MB
• Add in your submission script (include ~10% extra):
export SCOREP_TOTAL_MEMORY=20MB

Overhead
• You need to declare enough size of SCOREP_TOTAL_MEMORY to avoid flushing of the
performance files.
• For our application, non instrumented execution on 1 node takes ~30 seconds, while for profiling
and tracing is 45 and 53 seconds respectively, so 50% and 76% overhead.
• The overhead always depends on the application and what you instrument, OpenMP etc.
• We have choice of selective instrumentation or manually profiling filter

How to use Scalasca/Score-P with tracing?
• In your submission script, replace:
scalasca -analyze jsrun …
with
scalasca -analyze -q -t jsrun
• The –q disables the profiling.

Initial view with tracing

Computation with tracing and expand trees

Late Sender for Point-to-Point communication

Late Sender - Time
Documentation:
https://apps.fz-juelich.de/scalasca/releases/scalasca/2.5/help/scalasca_patterns.html

Late Sender – Wrong Order Time/Different Sources

Late Sender – Wrong Order Time/Same source

MPI Wait at N x N Time

MPI Wait at N x N Time - Explanation

Number of MPI communications

Short and Long-term delay

Long-term delay sender

Propagating wait states

Tracing start

Terminal wait states

Direct wait stats

Wait States

Imbalance in the critical path

Critical Path Profile

Critical Path Imbalance

Intra-partition Imbalance

Non Critical Path Activities

MiniWeather – MPI+OMP
• Compile:
MPI: make PREP="scorep --mpp=mpi –openmp --pdt" openmp
• Execution (submission script):
scalasca -analyze -q -t jsrun -n 16 -r 2 -a 1 -c 8 "-b packed:8"
./miniWeather_mpi_openmp
• Visualize:
scalasca -examine /gpfs/…/
scorep_miniWeather_mpi_openmp_2p16x8_trace

OpenMP Views

OpenMP Thread Management Time

Duration of Fork

Source code of corresponding OpenMP call

OpenMP Thread Team Fork

Implicit Synchronization

Implicit Synchronization - Explanation

Idle threads

Idle threads - Explanation

Limited Parallelism – Process x Thread

Limited Parallelism - Explanation

Long-term delay costs

Long/Short-term OpenMP Thread Idleness delay costs

Computational Imbalance overload

Computational Imbalance overload – Process x Thread

Performance Analysis with Scalasca on Summit Supercomputer part I

More Related Content

What's hot (10)

Similar to Performance Analysis with Scalasca on Summit Supercomputer part I (20)

More from George Markomanolis (14)

Recently uploaded (20)

Performance Analysis with Scalasca on Summit Supercomputer part I