Analytical Models of Parallel Programs
Download as PPT, PDF17 likes3,305 views
This document discusses analytical models for parallel programs. It covers sources of overhead in parallel programs like load imbalance and dependencies. It defines performance metrics like execution time and overhead functions. It discusses how granularity affects performance and the scalability of parallel systems. It provides examples of optimizing dense matrix algorithms like matrix-vector and matrix-matrix multiplication. The goal is to determine minimum execution times and costs by analyzing parallel runtimes as a function of problem size and number of processors.
Analytical Models of Parallel Programs
- 1. ANALYTICAL MODELS OF PARALLEL PROGRAMS Prof. Shashikant V. Athawale Assistant Professor | Computer Engineering Department | AISSMS College of Engineering, Kennedy Road, Pune , MH, India - 411001
- 2. Contents ❖ Analytical Models: Sources of overhead in Parallel Programs. ❖ Performance Metrics for Parallel Systems ❖ The effect of Granularity on Performance ❖ Scalability of Parallel Systems ❖ Minimum execution time and minimum cost ❖ Optimal execution time. ❖ Dense Matrix Algorithms: Matrix-Vector Multiplication ❖ Matrix-Matrix Multiplication.
- 3. Basics of Analytical Modelling ❖ A sequential algorithm is evaluated by its runtime in function of its input size. ➢ O(f(n)), Ω(f(n)), Θ(f(n)). ❖ The asymptotic runtime is independent of the platform. Analysis “at a constant factor”. ❖ A parallel algorithm is evaluated by its runtime in function of ➢ the input size ➢ the number of processors, ➢
- 4. Sources of overhead in parallel programs ❖ Overheads: wasted computation, communication, idling, contention. ➢ Inter-process interaction. ➢ Load imbalance. ➢ Dependencies.
- 5. Performance metrics for parallel systems ❖ Execution time = time elapsed between ➢ beginning and end of execution on a sequential computer. ➢ beginning of first processor and end of the last processor on a parallel computer.
- 6. Performance metrics for parallel systems ❖ Total parallel overhead. ➢ Total time collectively spent by all processing elements = pTp ➢ Time spent doing useful work (serial time) = Ts ➢ Overhead function: To = pTp-Ts.
- 7. Effect of Granularity on Performance ❖ Scaling down: To use fewer processing elements than the maximum possible. ❖ Naïve way to scale down: ➢ Assign the work of n/p processing element to every processing element. ■ Computation increases by n/p. ■ Communication growth ≤ n/p. ❖ If a parallel system with n processing elements is cost optimal, then it is still cost optimal with p.
- 8. Scalability of Parallel Systems ❖ Scalability: ability to use efficiently increasing processing power. ❖ Can maintain its efficiency constant when increasing the number of processors and the size of the problem. ❖ In many cases T0=f(TS,p) and grows sub-linearly with TS. It can be possible to increase p and TS and keep E constant. ❖ Scalability measures the ability to increase speedup in function of p.
- 9. Minimum execution time and minimum cost ❖ We can determine the minimum parallel runtime T min P for a given W by differentiating the expression for TP w.r.t. p and equating it to zero. ❖ If p0 is the value of p as determined by this equation, Tp (p0) is the minimum parallel time.
- 10. Dense Matrix Algorithms: Matrix-Vector Multiplication ❖ We aim to multiply a dense n × n matrix A with an n × 1 vector x to yield the n × 1 result vector y. ❖ The serial algorithm requires n2 multiplications and additions. W = n2
- 11. Matrix Vector Multiplication ❖ There are Two Types : ➢ Rowwise 1-D Partitioning ■ The n × n matrix is partitioned among n processors, with each processor storing complete row of the matrix. ■ The n × 1 vector x is distributed such that each process owns one of its elements.
- 12. Matrix Vector Multiplication ❖ 2-D partitioning ■ The n × n matrix is partitioned among n2 processors such that each processor owns a single element. ■ The n × 1 vector x is distributed only in the last column of n processors.
- 13. Matrix-Matrix Multiplication ❖ The problem of multiplying two n × n dense, square matrices A and B to yield the product matrix C = A × B. ❖ The serial complexity is O(n3 ).