Chapter 3 pc

Chapter 3Chapter 3
Principles of Parallel AlgorithmPrinciples of Parallel Algorithm
DesignDesign
Dr. Muhammad Hanif Durad
Department of Computer and Information Sciences
Pakistan Institute Engineering and Applied Sciences
hanif@pieas.edu.pk
Some slides have bee adapted with thanks from DANA
PETCU'S at http://web.info.uvt.ro/~petcu/index.html

Dr. Hanif Durad 2
Lecture Outline (1/3)
 Introduction to Parallel Algorithms
 Tasks and Decomposition
 Processes and Mapping
 Processes Versus Processors
 Decomposition Techniques
 Recursive Decomposition
 Recursive Decomposition
 Exploratory Decomposition
 Hybrid Decomposition

Dr. Hanif Durad 3
 Characteristics of Tasks and Interactions
 Task Generation, Granularity, and Context
 Characteristics of Task Interactions
 Mapping Techniques for Load Balancing
 Static and Dynamic Mapping

Dr. Hanif Durad 4
 Methods for Minimizing Interaction Overheads
 Maximizing Data Locality
 Minimizing Contention and Hot-Spots
 Overlapping Communication and Computations
 Replication vs. Communication
 Group Communications vs. Point-to-Point Communication
 Parallel Algorithm Design Models
 Data-Parallel, Work-Pool, Task Graph, Master-Slave,
Pipeline, and Hybrid Models

3.1 Preliminaries
Decomposition, Tasks, and
Dependency Graphs

3.1.1.1 Decomposition
 The process of dividing a computation into smaller
parts, some or all of which may potentially be executed
in parallel,
 Tasks are programmer-defined units of computation into
which the main computation is subdivided by means of
decomposition.
 Simultaneous execution of multiple tasks is the key to
reducing the time required to solve the entire problem.
 Tasks can be of arbitrary size, but once defined, they are
regarded as indivisible units of computation.
 The tasks into which a problem is decomposed may not
all be of the same size.

Example: Multiplying a Dense
Matrix with a Vector (2/2)
 Computation of each element of output vector y is
independent of other elements. Based on this, a
dense matrix-vector product can be decomposed
into n tasks. The figure highlights the portion of the
matrix and vector accessed by Task 1.
 Observations: While tasks share data (namely, the
vector b ), they do not have any control
dependencies - i.e., no task needs to wait for the
(partial) completion of any other. All tasks are of the
same size in terms of number of operations.
 Is this The maximum number of tasks we could
decompose this problem into? No= can be n2

Dr. Hanif Durad 8
Example 3.1: Multiplying a Dense
Matrix with a Vector (1/2)
n x n n x 1 n x 1
n =12

3.1.1.2 Tasks Dependency
Graphs
 Some tasks may use data produced by other tasks and thus may
need to wait for these tasks to finish execution.
 It is an abstraction used to express such dependencies among tasks
and their relative order of execution
 A task-dependency graph is a directed acyclic graph in which the nodes
represent tasks and the directed edges indicate the dependencies amongst
them
 The task corresponding to a node can be executed when all tasks connected to
this node by incoming edges have completed.
 Task-dependency graphs can be disconnected & the edge-set of a task-
dependency graph can be empty.

The case for matrix-vector x, where each task computes a subset of the entries of
the product vector.
 The nodes with no incoming edges are start nodes and the nodes
with no outgoing edges are finish nodes.

Dr. Hanif Durad 10
Processing
Consider the execution of the query:
MODEL = ``CIVIC'' AND YEAR = 2001 AND
(COLOR = ``GREEN'' OR COLOR = ``WHITE)
on the following database:
(1/9)

Dr. Hanif Durad 11
ID# Model Year Color Dealer Price
4523 Civic 2002 Blue MN $18,000
3476 Corolla 1999 White IL $15,000
7623 Camry 2001 Green NY $21,000
9834 Prius 2001 Green CA $18,000
6734 Civic 2001 White OR $17,000
5342 Altima 2001 Green FL $19,000
3845 Maxima 2001 Blue NY $22,000
8354 Accord 2000 Green VT $18,000
4395 Civic 2001 Red CA $17,000
7352 Civic 2002 Red WA $18,000
(2/9)

Dr. Hanif Durad 12
Example: Database Query
Processing (3/9)
 The execution of the query can be divided into
subtasks in various ways. Each task can be
thought of as generating an intermediate table
of entries that satisfy a particular clause.

Dr. Hanif Durad 13
• Decomposing the given query into a number of tasks. Edges in this
graph denote that the output of one task is needed to accomplish the
next.
(4/9)

Dr. Hanif Durad 14
Processing (5/9)
 Note that the same problem can be
decomposed into subtasks in other ways as
well

Dr. Hanif Durad 15
An alternate decomposition of the given problem into subtasks, along
with their data dependencies
(6/9)

Dr. Hanif Durad 16
Processing (7/9)
 Different task decompositions may lead to
significant differences with respect to their
eventual parallel performance.
 Note that the same problem can be
decomposed into subtasks in other ways as
well.

Dr. Hanif Durad 17
An alternate decomposition of the given problem into subtasks,
along with their data dependencies
(8/9)

Dr. Hanif Durad 18
Processing (9/9)
 Different task decompositions may lead to
significant differences with respect to their
eventual parallel performance.

Dr. Hanif Durad 19
3.1.2.1 Granularity
 The number and size of tasks into which a problem is
decomposed
 determines the granularity of the decomposition.
 A decomposition into a large number of small tasks is called
finegrained
 A decomposition into a small number of large tasks is called
coarse-grained.
 Example:
 First decomposition for matrix-vector multiplication is fine-grained
 Second decomposition is a coarse-grained decomposition of the same
problem into four tasks (next page)

Dr. Hanif Durad 20
A coarse grained counterpart to the dense matrix-vector product
example. Each task in this example corresponds to the computation
of three elements of the result vector.
n/3 =4

Dr. Hanif Durad 21
3.1.2.3 Maximum degree of
concurrency
 Maximum no. of tasks executed simultaneously in a
parallel program at any given time.
 It is usually less than the total no. tasks due to
dependencies among the tasks.
 Ex.: for task-dependency graphs that are trees, equal to
the no. leaves in the tree.
 What is the maximum degree of concurrency of the
database query examples?
Ans= 4

Dr. Hanif Durad 22
3.1.2.4 Average degree of
concurrency
 the average number of tasks that can run concurrently
over the entire duration of execution of the program.
 Example:
First decomposition of matrix-vector multiplication has a
fairly small granularity and a large degree of
concurrency.
Second decomposition for the same problem has a larger
granularity and a smaller degree of concurrency.

Dr. Hanif Durad 23
Average degree of concurrency=3
Max. degree of concurrency=4

Dr. Hanif Durad 24
Average degree of concurrency=3
Max. degree of concurrency=4

Dr. Hanif Durad 25
3.1.2.5 Tasks Dependency
Graphs and Concurrency (1/2)
 Degree of concurrency depends on the shape of the
task-dependency graph
 A feature of a task-dependency graph that
determines the average degree of concurrency for
a given granularity is its critical path
 The longest directed path between any pair of start
and finish nodes: the critical path.

Dr. Hanif Durad 26
Tasks Dependency Graphs and
Concurrency (2/2)
 The sum of the weights of nodes along this path is
the critical path length, where the weight of a node
is the size or the amount of work associated with
the corresponding task.
 Ratio of the total amount of work to the critical-
path length: average degree of concurrency
 A shorter critical path favors a higher degree of
concurrency.

Dr. Hanif Durad 27
Critical Path Length
Consider the task dependency graphs of the two
database query decompositions:

Dr. Hanif Durad 28
Critical Path Length
weight of a node is the size or the amount of work

Dr. Hanif Durad 29
Critical Path Length (CPL)
 What are the critical path lengths for the two task
dependency graphs?
Ans= 27 and 34

Dr. Hanif Durad 30
(CPL)
CPL=27 CPL=34

Amount of Work (AW)
 Total amount of work required to solve the
problems using the two decompositions ?
 Fig (b)
AW=10x4+6+11+7
=64
 Fig (a)
AW=10x4+9+6+8
=63

Average Degree of Concurrency
(ADC)
 The average degree of concurrency of the
two task-dependency graphs ?
 Fig (b)
ADC = AW/ CPL
=64/34
=1.88
 Fig (a)
ADC = AW/ CPL
=63/27
=2.33
 Which graph to choose ?

Dr. Hanif Durad 33
Shortest Parallel Execution
 If each task takes 10 time units.
 What is the shortest parallel execution time for each
decomposition?
Ans= 30 and 40 units

Dr. Hanif Durad 34
How many processors are needed
 How many processors are needed in each case to
achieve this minimum parallel execution time?
Ans= 4

Dr. Hanif Durad 35
Maximum Degree of
Concurrency (MDC)
 What is the maximum degree of concurrency?
Ans= 4

Dr. Hanif Durad 36
3.1.2.6 Limitation Factors (1/2)
 Usually, there is an inherent bound on how fine-
grained a decomposition a problem permits.
 For instance, there are n2
multiplications and
additions in matrix-vector multiplication considered
in the example and the problem cannot be
decomposed into more than O(n2) tasks even by
using the most finegrained decomposition.

Limitation Factors (2/2)
 The interaction among tasks running on different
physical processors.
 The tasks that a problem is decomposed into often share
input, output, or intermediate data.
 The dependencies in a task-dependency graph usually
result from the fact that the output of one task is the input
for another.
 Example: in the database query example, tasks share
intermediate data; the table generated by one task is often
used by another task as input.

Dr. Hanif Durad 38
3.1.2.7 Task Interaction Graphs
 Depending on the definition of the tasks and the
parallel programming paradigm, there may be
interactions among tasks that appear to be independent
in a task-dependency graph.
 Example, in the decomposition for matrix-vector
multiplication, although all tasks are independent, they all
need access to the entire input vector b.

Since originally there is only one copy of b, tasks may have to send
and receive messages for all of them to access the entire vector in the
distributed-memory paradigm.

Task Interaction Graphs
 Captures the pattern of interaction among tasks
 The nodes in a task-interaction graph represent tasks and the edges
connect tasks that interact with each other.
 The nodes and edges of a task-interaction graph can be assigned
weights proportional to the amount of computation a task performs
and the amount of interaction that occurs along an edge, if this
information is known.
 The edges in a task-interaction graph are usually undirected, but
directed edges can be used to indicate the direction of flow of data,
if it is unidirectional.
 The edge-set of a task-interaction graph is usually a superset of the
edge-set of the task-dependency graph.
 Example: in the database query example, the task-interaction graph is the
same as the task-dependency graph.

Task Interaction Graphs: An
Example (1/3)
 Consider: ? the product y = Ab of a sparse n x n matrix A with a
dense nx1 vector b.
 A matrix is considered sparse when a significant no. entries in it
are zero and the locations of the non-zero entries do not conform
to a predefined structure or pattern.
 Arithmetic operations involving sparse matrices can often be
optimized significantly by avoiding computations involving the
zeros.
 While computing the ith entry of the product vector,
we need to compute the products for only those values
of j for which not equal with 0.
 For example, y[0] = A[0, 0].b[0] + A[0, 1].b[1] + A[0, 4].b[4] + A[0,
8].b[8].
y[i]= (A[i,j]b[j])1
n
j∑ =
A[i,j]b[j]
A[i,j]

Task Interaction Graphs: An
Example (2/3)
 One possible way of decomposing this computation is to partition the
output vector y and have each task compute an entry in it (Fig. (a))
 Assigning the computation of the element y[i] of the output vector to Task i,
 Task i is the "owner" of row A[i, *] of the matrix and the element b[i] of the
input vector.
 The computation of y[i] requires access to many elements of b that are owned
by other tasks
 So Task i must get these elements from the appropriate locations.

In the message-passing paradigm, with the ownership of b[i],Task i also inherits the
responsibility of sending b[i] to all the other tasks that need it for their computation.

For example: Task 4 must send b[4] to Tasks 0, 5, 8, and 9 and must get b[0], b[5],
b[8], and b[9] to perform its own computation.

Dr. Hanif Durad 42
Task Interaction Graphs: An Example
(3/3)

Dr. Hanif Durad 43
 Task interaction graphs represent data dependences
 Task dependency graphs represent control
dependences
Task Interaction Graphs vs. Task
Dependency Graphs

Dr. Hanif Durad 44
3.1.3 Processes and Mapping
(1/5)
 In general, the number of tasks in a decomposition
exceeds the number of processing elements
available.
 For this reason, a parallel algorithm must also
provide a mapping of tasks to processes.

Dr. Hanif Durad 45
Processes and Mapping (2/5)
Note: We refer to the mapping as being from tasks
to processes, as opposed to processors. This is
because typical programming APIs, as we shall see,
do not allow easy binding of tasks to physical
processors. Rather, we aggregate tasks into
processes and rely on the system to map these
processes to physical processors. We use processes,
not in the UNIX sense of a process, rather, simply as
a collection of tasks and associated data.

Dr. Hanif Durad 46
 Appropriate mapping of tasks to processes is critical to the
parallel performance of an algorithm.
 Mappings are determined by both the task dependency and task
interaction graphs.
 Task dependency graphs can be used to ensure that work is
equally spread across all processes at any point (minimum idling
and optimal load balance).
 Task interaction graphs can be used to make sure that processes
need minimum interaction with other processes (minimum
communication).

Dr. Hanif Durad 47
An appropriate mapping must minimize parallel
execution time by:
 Mapping independent tasks to different processes.
 Assigning tasks on critical path to processes as
soon as they become available.
 Minimizing interaction between processes by
mapping tasks with dense interactions to the same
process.

 Difficulty: criteria often conflict with one
another
 e.g. no decomposition minimizes interactions but no
speedup!
Dr. Hanif Durad 48

Dr. Hanif Durad 49
Processes and Mapping: Example
(1/2)
 Mapping tasks in the database query
decomposition to processes.
 These mappings were arrived at by viewing
the dependency graph in terms of levels (no
two nodes in a level have dependencies).
 Tasks within a single level are then assigned
to different processes.

Dr. Hanif Durad 50
Processes and Mapping: Example
(2/2)

3.1.4 Processes vs. processors
(1/2)
 Processors are physical resources
 Processes provide a convenient way of abstracting or
virtualizing a multiprocessor.
 Write parallel programs in terms of processes not
physical processors;
 Mapping processes to processors is a subsequent step.
 The number of processes does not have to be the same
as the number of processors available to the program.
 If there are more processes, they are multiplexed onto the
available processors;
 If there are fewer processes, then some processors will remain
idle

Processes vs. processors (2/2)
 The correct OS definition of a process is that of an
address space and one or more threads of control that
share that address space.
 Thus, processes and threads are distinguished in that
definition.
 In what follows: assume that a process has only one
thread of control.

Dr. Hanif Durad 53
3.2 Decomposition Techniques
 So how does one decompose a task into various
subtasks?
 While there is no single recipe that works for all
problems, we present a set of commonly used
techniques that apply to broad classes of
problems.

Dr. Hanif Durad 54
Decomposition Techniques
include:
 recursive decomposition
 data decomposition
 Partitioning output data
 Partitioning input data
 Partitioning input/output data
 Partitioning intermediate data
 exploratory decomposition
 speculative decomposition
 Hybrid Decomposition

Decomposition Techniques
 The recursive- and data-decomposition
techniques are relatively general purpose as they
can be used to decompose a wide variety of
problems.
 Speculative- and exploratory-decomposition
techniques are more of a special purpose nature
because they apply to specific classes of
problems
Dr. Hanif Durad 55

3.2.1 Recursive Decomposition
(1/2)
 Is a method for inducing concurrency in problems that
can be solved using the divide-and-conquer strategy.
 A problem is solved by first dividing it into a set of
independent subproblems.
 Each one of these subproblems is solved by recursively
applying a similar division into smaller subproblems followed
by a combination of their results.
 The divide-and-conquer strategy results in natural
concurrency, as different subproblems can be solved
concurrently.
(1/10)

Recursive Decomposition (2/2)
 Example: Quicksort.
 Sorting a sequence A of n elements using the
commonly used quicksort algorithm.
 We define a task as the work of partitioning a given
subsequence.
 Fig also represents the task graph for the problem.
(2/10)

Recursive Decomposition: Example-1
(1/2)
(3/10)

Dr. Hanif Durad 59
Recursive Decomposition:
Example-1 (2/2)
In this example, once the list has been partitioned
around the pivot, each sublist can be processed
concurrently (i.e., each sublist represents an
independent subtask). This can be repeated
recursively.
(4/10)

finding the minimum (1/3)
 Sometimes, it is possible to restructure a computation to
make it amenable to recursive decomposition even if the
commonly used algorithm for the problem is not based on
the divide-and-conquer strategy.
 Example: the problem of finding the minimum element in an
unordered sequence A of n elements.
 The serial algorithm for solving this problem scans the entire
sequence A, recording at each step the minimum element found
so far as illustrated in the following serial algorithm.
 Once we restructure this computation as a divide-and-conquer
algorithm, we can use recursive decomposition to extract
concurrency.
 A recursive program for finding the minimum in an array of
numbers A of length n is shown next
(5/10)

Dr. Hanif Durad 61
Recursive Decomposition: Example
1. procedure SERIAL_MIN (A,n)
2. begin
3. min = A[0];
4. for i:= 1 to n−1 do
5. if (A[i] <min) min := A[i];
6. endfor;
7. return min;
8. end SERIAL_MIN
(6/10)

Example
We can rewrite the loop as follows:
1. procedure RECURSIVE_MIN (A, n)
2. begin
3. if ( n = 1 ) then
4. min := A [0] ;
5. else
6. lmin := RECURSIVE_MIN ( A, n/2 );
7. rmin := RECURSIVE_MIN ( &(A[n/2]), n - n/2 );
8. if (lmin < rmin) then
9. min := lmin;
10. else
11. min := rmin;
12. endelse;
13. endelse;
14. return min;
15. end RECURSIVE_MIN
(7/10)

 In this algorithm
 A is split into two subsequences, each of size n/2,
 find the minimum for each of these subsequences by
performing a recursive call.
 Now the overall minimum element is found by
selecting the minimum of these two subsequences.
 The recursion terminates when there is only one
element left in each subsequence.
(8/10)

 It is easy to construct a task dependency graph for this
problem.
 Fig. illustrates a task-dependency graph for finding the
minimum of eight numbers where each task is assigned
the work of finding the minimum of two numbers.
 The task-dependency graph for finding the minimum
number in the sequence {4, 9, 1, 7, 8, 11, 2, 12}.
 Each node in the tree represents the task of finding the
minimum of a pair of numbers.
(9/10)

Dr. Hanif Durad 65
Example
(10/10)

Dr. Hanif Durad 66
3.2.2 Data Decomposition (1/3)
 Identify the data on which computations are
performed.
 Partition this data across various tasks.
 This partitioning induces a decomposition of
the problem.
 Data can be partitioned in various ways - this
critically impacts performance of a parallel
algorithm.

3.2.2 Data Decomposition (2/3)
 Is a powerful and commonly used method for
deriving concurrency in algorithms that operate
on large data structures.
 The decomposition of computations is done in
two steps.
1. The data on which the computations are performed
is partitioned,
2. This data partitioning is used to induce a partitioning
of the computations into tasks.

Data Decomposition (3/3)
 The operations that these tasks perform on
different data partitions are usually similar (e.g.,
matrix multiplication that follows) or are chosen
from a small set of operations (e.g., LU
factorization).
 One must explore and evaluate all possible ways
of partitioning the data and determine which one
yields a natural and efficient computational
decomposition.

3.2.2.1 Partitioning Output Data
(1/10)
 In many computations, each element of the
output can be computed independently of others
as a function of the input.
 In such computations, a partitioning of the output
data automatically induces a decomposition of
the problems into tasks
 each task is assigned the work of computing a
portion of the output

Partitioning Output Data: Example-1
Consider the problem of multiplying two n x n matrices A
and B to yield matrix C. The output matrix C can be
partitioned into four tasks as follows:
(2/10)
(1/3)

Dr. Hanif Durad 71
Partitioning Output Data :
Example-1 (2/3)
The 4 sub-matrices of C, roughly of size
(n/2 x n/2) each, are then independently
computed by 4 tasks as the sums of the
appropriate products of sub-matrices of A and
B
 Other task decompositions possible
(3/10)

Partitioning Output Data: Example-1
Decomposition I Decomposition II
Task 1: C1,1 = A1,1 B1,1
Task 2: C1,1 = C1,1 + A1,2 B2,1
Task 3: C1,2 = A1,1 B1,2
Task 4: C1,2 = C1,2 + A1,2 B2,2
Task 5: C2,1 = A2,1 B1,1
Task 6: C2,1 = C2,1 + A2,2 B2,1
Task 7: C2,2 = A2,1 B1,2
Task 8: C2,2 = C2,2 + A2,2 B2,2
Task 1: C1,1 = A1,1 B1,1
Task 2: C1,1 = C1,1 + A1,2 B2,1
Task 3: C1,2 = A1,2 B2,2
Task 4: C1,2 = C1,2 + A1,1 B1,2
Task 5: C2,1 = A2,2 B2,1
Task 6: C2,1 = C2,1 + A2,1 B1,1
Task 7: C2,2 = A2,1 B1,2
Task 8: C2,2 = C2,2 + A2,2 B2,2
 Independent task ?
(4/10)
(3/3)

Example -2 (1/6)
 Computing frequencies of itemsets in a
transaction database.
 Given a set T containing n transactions and a set I
containing m itemsets.
 Each transaction and itemset contains a small no.
items, out of a possible set of items.
(5/10)

Example -2 (2/6)
 Example: T is a grocery stores database of
customer sales with each transaction being an
individual grocery list of a shopper & itemset
could be a group of items in the store.
 If the store desires to find out how many customers
bought each of the designated groups of items, then it
would need to find the no. times that each itemset in I
appears in all the transactions (the no. transactions of
which each itemset is a subset of)
Dr. Hanif Durad 74
(6/10)

Dr. Hanif Durad 75
Partitioning Output Data : Example-2
(7/10)
(3/6)

Dr. Hanif Durad 76
Partitioning Output Data : Example -2
(8/10)
(4/6)

Example -2 (5/6)
 Fig. (a) shows an example
 The database shown consists of 10 transactions,
and
 We are interested in computing the frequency of the 8
itemsets shown in the second column.
 The actual frequencies of these itemsets in the
database (output) are shown in the third column.
 For instance, itemset {D, K} appears twice, once in
the second and once in the ninth transaction.
(9/10)

Example -2 (6/6)
 Fig. (b) shows how the computation of
frequencies of the itemsets can be decomposed
into 2 tasks
 by partitioning the output into 2 parts and having each
task compute its half of the frequencies.
Dr. Hanif Durad 78
(10/10)

3.2.2.2 Partitioning Input Data
(1/6)
 Remark: Partitioning of output data can be performed
only if each output can be naturally computed as a
function of the input.
 In many algorithms, it is not possible or desirable to
partition the output data.
 For example:
 While finding the minimum, maximum, or the sum of a set of
numbers, the output is a single unknown value.
 In a sorting algorithm, the individual elements of the output
cannot be efficiently determined in isolation.

Dr. Hanif Durad 80
Partitioning Input Data (2/6)
 It is sometimes possible to partition the input data, and
then use this partitioning to induce concurrency.
 A task is created for each partition of the input data and this
task performs as much computation as possible using these
local data.
 Solutions to tasks induced by input partitions may not
directly solve original problem

In such cases, a follow-up computation is needed to combine the
results.

Dr. Hanif Durad 81
Partitioning Input Data
Example -1
 Example: finding the sum of N numbers using p
processes (N > p):
 We can partition the input into p subsets of nearly equal
sizes.
 Each task then computes the sum of the numbers in one of
the subsets.
 Finally, the p partial results can be added up to yield the
final result
(3/6)

Example -2 (1/3)
 The problem of computing the frequency of a
set of itemsets
 Can also be decomposed based on a partitioning
of input data.
 Fig. shows a decomposition based on a
partitioning of the input set of transactions.
Dr. Hanif Durad 82
(4/6)

Dr. Hanif Durad 83
Partitioning Input Data : Example-2
Intermediate
result
Intermediate
result
(2/3)
(5/6)

Example -2 (3/3)
 Each of the two tasks computes the frequencies
of all the itemsets in its respective subset of
transactions.
 The two sets of frequencies, which are the
independent outputs of the two tasks, represent
intermediate results.
 Combining the intermediate results by pair wise
addition yields the final result.
84
(6/6)

3.2.2.3 Partitioning both Input
and Output Data (1/2)
 In some cases, in which it is possible to partition the
output data, partitioning of input data can offer
additional concurrency.
 Example: consider the 4-way decomposition shown in
Fig. for computing itemset frequencies.
 both the transaction set and the frequencies are divided into
two parts and a different one of the four possible combinations
is assigned to each of the four tasks.
 Each task then computes a local set of frequencies.
 Finally, the outputs of Tasks 1 and 3 are added together, as are
the outputs of Tasks 2 and 4.

Dr. Hanif Durad 86
Partitioning Input and Output Data
(2/2)

Dr. Hanif Durad 87
3.2.2.4 Intermediate Data
Partitioning (1/8)
 Partitioning intermediate data can sometimes lead to
higher concurrency than partitioning input or output
data.
 Often, the intermediate data are not generated explicitly
in the serial algorithm for solving the problem
 Some restructuring of the original alg. May be required
to use intermediate data partitioning to induce a
decomposition

Intermediate Data Partitioning
(2/8)
 Example: matrix multiplication
 Recall that the decompositions induced by a 2x 2
partitioning of the output matrix C have a maximum
degree of concurrency of four.
 Increase the degree of concurrency by introducing an
intermediate stage in which 8 tasks compute their
respective product submatrices and store the results
in a temporary 3-d matrix D, as shown in Fig.
 Submatrix Dk,i,j is the product of Ai,kand Bk,j.

Dr. Hanif Durad 89
Intermediate Data Partitioning: (3/8)
Example

(4/8)
 A partitioning of the intermediate matrix D
induces a decomposition into eight tasks.
 After the multiplication phase, a relatively
inexpensive matrix addition step can compute
the result matrix C
 All submatrices D*,i,jwith the same second and
third dimensions i and j are added to yield Ci,j.

Dr. Hanif Durad 91
Intermediate Data Partitioning:
Example (5/8)
A decomposition of intermediate data structure leads
to the following decomposition into 8 + 4 tasks:
Stage II
Stage I

(6/8)
 The eight tasks numbered 1 through 8 in Fig.
perform O(n3/8) work each in multiplying (n/2 x
n/2 ) submatrices of A and B
 Then, four tasks numbered 9 through 12 spend
O(n2/4) time each in adding the appropriate (n/2 x
n/2) submatrices of the intermediate matrix D to
yield the final result matrix C
 Second Fig. shows the task dependency graph

Dr. Hanif Durad 93
Example (7/8)
Task 01: D1,1,1= A1,1 B1,1 Task 02: D2,1,1= A1,2 B2,1
Task 03: D1,1,2= A1,1 B1,2 Task 04: D2,1,2= A1,2 B2,2
Task 05: D1,2,1= A2,1 B1,1 Task 06: D2,2,1= A2,2 B2,1
Task 07: D1,2,2= A2,1 B1,2 Task 08: D2,2,2= A2,2 B2,2
Task 09: C1,1 = D1,1,1 + D2,1,1 Task 10: C1,2 = D1,1,2 + D2,1,2
Task 11: C2,1 = D1,2,1 + D2,2,1 Task 12: C2,,2 = D1,2,2 + D2,2,2

Dr. Hanif Durad 94
Example (8/8)
The task dependency graph for the
decomposition (shown in previous foil) into 12
tasks is as follows:

The Owner Computes Rule (1/2)
 A decomposition based on partitioning output or input
data is also widely referred to as the owner-computes
rule.
 The idea behind this rule is that each partition performs
all the computations involving data that it owns.
 Depending on the nature of the data or the type of data-
partitioning, the owner-computes rule may mean
different things

The Owner Computes Rule (2/2)
 When we assign partitions of the input data to
tasks, then the owner computes rule means that a
task performs all the computations that can be
done using these data.
 If we partition the output data, then the owner-
computes rule means that a task computes all the
data in the partition assigned to it.

3.2.3 Exploratory Decomposition
(1/10)
 In many cases, the decomposition of the problem
goes hand-in-hand with its execution.
 These problems typically involve the exploration
(search) of a state space of solutions.
 Problems in this class include a variety of
discrete optimization problems (0/1 integer
programming, QAP (quadratic assignment
problem), etc.), theorem proving, game playing,
etc.

Exploratory Decomposition-
Example-1=> 15-puzzle problem
 Consists of 15 tiles numbered 1 through 15 and one blank
tile placed in a 4 x 4 grid
 A tile can be moved into the blank position from a
position adjacent to it, thus creating a blank in the tile's
original position.
 Four moves are possible: up, down, left, and right.
 The initial and final configurations of the tiles are
specified.
(2/10)

Example-1 =>15-puzzle problem (1/4)
 Objective is to determine any sequence or a shortest
sequence of moves that transforms the initial configuration
to the final configuration
 Fig. illustrates sample initial and final configurations and a
sequence of moves leading from the initial configuration to
the final configuration
(3/10)

 The puzzle is typically solved using tree-search techniques.
 From initial configration, all possible successor configurations
are generated.
 May have 2, 3, or 4 possible successor configurations depending
occupation of the empty slot by one of its neighbors.
 The task of finding a path from initial to final configuration now
translates to finding a path from one of these newly generated
configurations to the final configuration.
 Since one of these newly generated configurations must be closer
to the solution by one move (if a solution exists), we have made
some progress towards finding the solution.
(4/10)

 The configuration space generated by the tree search is the state space
graph.
 Each node of the graph is a configuration and each edge of the graph connects
configurations that can be reached from one another by a single move of a tile.
 One method for solving this problem in parallel:
 First, a few levels of configurations starting from the initial configuration are
generated serially until the search tree has a sufficient number of leaf nodes
 Now each node is assigned to a task to explore further until at least one of them
finds a solution
 As soon as one of the concurrent tasks finds a solution it can inform the others
to terminate their searches.
 Figure illustrates one such decomposition into four tasks in which task 4 finds
the solution
(5/10)

(6/10)

Exploratory vs. data
decomposition (7/10)
 The tasks induced by data-decomposition are performed
in their entirety & each task performs useful
computations towards the solution of the problem
 In exploratory decomposition, unfinished tasks can be
terminated as soon as an overall solution is found.
 Portion of the search space searched (& the aggregate amount
of work performed) by a parallel formulation can be different
from that searched by a serial algorithm
 The work performed by the parallel formulation can be either
smaller or greater than that performed by the serial algorithm.

Example-2 (1/3) (8/10)
 Example: consider a search space that has been partitioned
into four concurrent tasks as shown in Fig.
 If the solution lies right at the beginning of the search
space corresponding to task 3
 (Fig. (a)), then it will be found almost immediately by the
parallel formulation.
 The serial algorithm would have found the solution only
after performing work equivalent to searching the entire
space corresponding to tasks 1 and 2.

Dr. Hanif Durad 105
Example-2 (2/3) (9/10)

Dr. Hanif Durad 106
 On the other hand, if the solution lies towards the
end of the search space corresponding to task 1
(Fig (b)), then the parallel formulation will
perform almost four times the work of the serial
algorithm and will yield no speedup.
 This change results in super- or sub-linear
speedups.
Example-2 (3/3) (10/10)

3.2.4 Speculative Decomposition
(1/9)
 In some applications, dependencies between tasks are not
known a-priori.
 For such applications, it is impossible to identify
independent tasks.
 When a program may take one of many possible
computationally significant branches depending on the
output of other computations that precede it.
 While one task is performing the computation whose output is
used in deciding the next computation, other tasks can
concurrently start the computations of the next stage.

Dr. Hanif Durad 108
 There are generally two approaches to dealing with
such applications: conservative approaches, which
identify independent tasks only when they are
guaranteed to not have dependencies, and, optimistic
approaches, which schedule tasks even when they may
potentially be erroneous.
 Conservative approaches may yield little concurrency
and optimistic approaches may require roll-back
mechanism in the case of an error.
Speculative Decomposition
(2/9)

(3/9)
 This scenario is similar to evaluating one or more of the branches
of a switch statement in C in parallel before the input for the
switch is available.
 While one task is performing the computation that will eventually resolve
the switch, other tasks could pick up the multiple branches of the switch in
parallel.
 When the input for the switch has finally been computed, the computation
corresponding to the correct branch would be used while that
corresponding to the other branches would be discarded.
 The parallel run time is smaller than the serial run time by the amount of
time required to evaluate the condition on which the next task depends
because this time is utilized to perform a useful computation for the next
stage in parallel.

 This parallel formulation of a switch guarantees at least some
wasteful computation.
 In order to minimize the wasted computation, a slightly
different formulation of speculative decomposition could be
used, especially in situations where one of the outcomes of
the switch is more likely than the others.
 In this case, only the most promising branch is taken up a task in
parallel with the preceding computation.
 In case the outcome of the switch is different from what was
anticipated, the computation is rolled back and the correct branch of
the switch is taken.
 The speedup due to speculative decomposition can add up if there are
multiple speculative stages

Dr. Hanif Durad 111
Speculative Decomposition:
Example-1
A classic example of speculative decomposition is
in discrete event simulation.
 The central data structure in a discrete event
simulation is a time-ordered event list.
 Events are extracted precisely in time order,
processed, and if required, resulting events are
inserted back into the event list.

Dr. Hanif Durad 112
Example-1
Consider your day today as a discrete event system - you
get up, get ready, drive to work, work, eat lunch, work
some more, drive back, eat dinner, and sleep.
 Each of these events may be processed independently,
however, in driving to work, you might meet with an
unfortunate accident and not get to work at all.
 Therefore, an optimistic scheduling of other events will
have to be rolled back.

Dr. Hanif Durad 113
Example-2
Another example is the simulation of a network
of nodes (for instance, an assembly line or a
computer network through which packets pass).
The task is to simulate the behavior of this
network for various inputs and node delay
parameters (note that networks may become
unstable for certain values of service rates,
queue sizes, etc.).

Dr. Hanif Durad 114
Example -2

Example-2
 The problem of simulating a sequence of input jobs on the
network described in this example appears inherently
sequential because the input of a typical component is the
output of another
 However, we can define speculative tasks that start simulating
a subpart of the network, each assuming one of several possible
inputs to that stage.
 When an actual input to a certain stage becomes available (as a
result of the completion of another selector task from a previous
stage),
 then all or part of the work required to simulate this input would have
already been finished if the speculation was correct,
 or the simulation of this stage is restarted with the most recent correct
input if the speculation was incorrect.

Speculative vs. exploratory
decomposition
 In exploratory decomposition
 the output of the multiple tasks originating at a branch is unknown.
 the serial algorithm may explore different alternatives one after the other,
because the branch that may lead to the solution is not known beforehand
=> the parallel program may perform more, less, or the same amount of
aggregate work compared to the serial algorithm depending on the location of
the solution in the search space.
 In speculative decomposition
 the input at a branch leading to multiple parallel tasks is unknown
 the serial algorithm would strictly perform only one of the tasks at a
speculative stage because when it reaches the beginning of that stage, it
knows exactly which branch to take.
=> a parallel program employing speculative decomposition performs more
aggregate work than its serial counterpart.

3.2.5 Hybrid Decompositions
 Decomposition techs are not exclusive, and can
often be combined together.
 Often, a computation is structured into multiple
stages and it is sometimes necessary to apply
different types of decomposition in different stages.

Hybrid Decompositions
Example 1: Finding the minimum.
 Example 1: while finding the minimum of a large
set of n numbers,
 a purely recursive decomposition may result in far
more tasks than the number of processes, P, available.
 An efficient decomposition would partition the input
into P roughly equal parts and have each task
compute the minimum of the sequence assigned to it.
 The final result can be obtained by finding the
minimum of the P intermediate results by using the
recursive decomposition shown in Fig

Dr. Hanif Durad 119
Hybrid Decompositions-Finding the
minimum.

Hybrid Decompositions-
Example 2: Quicksort in parallel.
 recursive decomposition have been used in quicksort.
 This formulation results in O(n) tasks for the problem of sorting a
sequence of size n.
 But due to the dependencies among these tasks and due to
uneven sizes of the tasks, the effective concurrency is quite
limited.
 For example, the first task for splitting the input list into two
parts takes O(n) time, which puts an upper limit on the
performance gain possible via parallelization.
 The step of splitting lists performed by tasks in parallel
quicksort can also be decomposed using the input
decomposition technique.
 The resulting hybrid decomposition that combines recursive
decomposition and the input data-decomposition leads to a

Dr. Hanif Durad 121
3.3 Characteristics of Tasks and
Interactions
Once a problem has been decomposed into
independent tasks, the characteristics of these
tasks critically impact choice and performance of
parallel algorithms.

Dr. Hanif Durad 122
3.3.1 Characteristics of Tasks
Relevant task characteristics include:
1. Task generation.
a) Static task generation
b) Dynamic task generation
2. Task sizes.
3. Knowledge of Task sizes
4. Size of data associated with tasks.

Dr. Hanif Durad 123
3.3.1.1.1 Static Task Generation
Concurrent tasks can be identified a-priori.
These can typically be decomposed using
data or recursive decomposition techniques
Examples:
Matrix operations, graph algorithms, image
processing applications, and other regularly
structured problems fall in this class.

Dr. Hanif Durad 124
3.3.1.1.2 Dynamic Task
Generation
Tasks are generated as we perform
computation.
These applications are typically decomposed
using exploratory or speculative decompositions.
Examples:
A classic example of this is in game playing -
each 15 puzzle board is generated from the
previous one.

Dr. Hanif Durad 125
3.3.1.2 Task Sizes
 Task sizes may be uniform (i.e., all tasks are
the same size) or non-uniform.
 Decomposition of matrix multiplication is
example of uniform task size
 Quick sort is an example of non-uniform
task size

3.3.1.3 Knowledge of Task Sizes
 Also influences the choice of mapping scheme is
knowledge of the task size.
 If the size of all the tasks is known, then this
information can often be used in mapping of tasks to
processes.
 For example, in the various decompositions for matrix
multiplication discussed so far, the computation time for
each task is known before the parallel program starts.
 On the other hand, the size of a typical task in the 15-
puzzle problem is unknown. We do not know a priori
how many moves will lead to the solution from a given
state.

Dr. Hanif Durad 127
3.3.1.4 Size of Data Associated
with Tasks (1/2)
 Another important characteristic of a task is the size of
data associated with it.
 The reason this is an important consideration for mapping
is that the data associated with a task must be available to
the process performing that task, and the size and the
location of these data may determine the process that can
perform the task without incurring excessive data-
movement overheads.

Size of Data Associated with
Tasks (2/2)
 Different types of data associated with a task may have different
sizes.
 e.g. the input data may be small but the output may be large, or vice versa.
 For example, the input to a task in the 15-puzzle problem may be
just one state of the puzzle.
 This is a small input relative to the amount of computation that may be
required to find a sequence of moves from this state to a solution state.
 In the problem of computing the minimum of a sequence
 the size of the input is proportional to the amount of computation, but the
output is just one number
 In the parallel formulation of the quick sort
 the size of both the input and the output data is of the same order as the
sequential time needed to solve the task.

Dr. Hanif Durad 129
3.3.2 Characteristics of Inter-
Task Interactions (1/2)
 In any nontrivial parallel algorithm, tasks need
to interact with each other to share data, work,
or synchronization information.
 Different parallel algorithms require different
types of interactions among concurrent tasks.
 The nature of these interactions makes them
more suitable for certain programming
paradigms and mapping schemes, and less
suitable for others.

Dr. Hanif Durad 130
Characteristics of Task
Interactions (2/2)
 The types of inter-task interactions can be
described along different dimensions, each
corresponding to a distinct characteristic of the
underlying computations.
 Static versus Dynamic Interactions
 Regular versus Irregular Interaction Patterns
 Read-only versus Read-Write Interactions
 One-way versus Two-way Interactions

Dr. Hanif Durad 131
3.3.2.1.1 Static Interactions
 In Static Interactions the tasks and their
interactions are known a-priori. These are
relatively simpler to code easily in the message-
passing paradigm
Example
 The decompositions for parallel matrix
multiplication presented earlier in this chapter
exhibit static inter-task interactions

3.3.2.1.2 Dynamic Interactions
 In dynamic interactions the timing or interacting tasks cannot be
determined a-priori. These interactions are harder to code,
especially, as we shall see, using message passing APIs.
Example- Consider solving the 15-puzzle problem
 Tasks are assigned different states to explore after an initial step that
generates the desirable number of states by applying breadth-first search on
the initial state.
 It is possible that a certain state leads to all dead ends and a task exhausts
its search space without reaching the goal state, while other tasks are still
busy trying to find a solution.
 The task that has exhausted its work can pick up an unexplored state from
the queue of another busy task and start exploring it. The interactions
involved in such a transfer of work from one task to another are dynamic.

3.3.2.2.1 Regular Interaction
Patterns (1/2)
 There is a definite pattern (in the graph sense) to the interactions
and these patterns can be exploited for efficient implementation.
Example-Image dithering
 The color of each pixel is determined as the weighted average of
its original value and the values of its neighboring pixels.
 We can easily decompose this computation, by breaking the image
into square regions and using a different task to dither each one of
these regions.
 Note that each task needs to access the pixel values of the region
assigned to it as well as the values of the image surrounding its
region.
 Thus, if we regard the tasks as nodes of a graph with an edge
linking a pair of interacting tasks, the resulting pattern is 2-D mesh

Dr. Hanif Durad 134
Regular Interaction Pattern:
Example (2/2)

3.3.2.2.2 Irregular Interaction
Patterns (1/2)
 Interactions lack well-defined topologies
 Sparse matrix-vector multiplication provides a good
example of irregular interaction
 In this decomposition, even though each task, by virtue of
the decomposition, knows a priori which rows of matrix A
it needs to access, without scanning the row(s) of A
assigned to it, a task cannot know which entries of vector b
it requires
 The reason is that the access pattern for b depends on the
structure of the sparse matrix A.

Dr. Hanif Durad 136
Irregular Interaction Pattern:
Example (2/2)

Dr. Hanif Durad 137
3.3.2.3 Read-only versus Read-
Write
 Interactions may be read-only or read-write.
 In read-only interactions, tasks just read data
items associated with other tasks.
 In read-write interactions tasks read, as well as
modily data items associated with other tasks.
 In general, read-write interactions are harder to
code, since they require additional
synchronization primitives.

Read-only Example
In the decomposition for parallel matrix multiplication
shown in, the tasks only need to read the shared input
matrices A and B.

Read-Write Example
 In read-write interactions, multiple tasks need to
read and write on some shared data.
 Consider the problem of solving the 15-puzzle.
 Used an exhaustive search to find a solution.
 Can be made more efficient if we use heuristic to provide a
relative approximate indication of the distance of each state from
the solution
 The states that need to be expanded further are stored in a
priority queue and giving preference to more promising states
 Priority queue constitutes shared data and tasks need both read
and write access to it

Dr. Hanif Durad 140
3.3.2.4 One-way versus Two-way
(1/4)
 In some interactions, the data or work needed by a task
or a subset of tasks is explicitly supplied by another task
or subset of tasks. Such interactions are called two-way
interactions and usually involve predefined producer and
consumer tasks.
 In other interactions, only one of a pair of
communicating tasks initiates the interaction and
completes it without interrupting the other one. Such an
interaction is called a one-way interaction.

One-way versus Two-way (2/4)
 All read-only interactions can be formulated as one-way.
 Read-write interactions can be either one-way or two-
way.
 The shared-address-space programming paradigms can
handle both one-way and two-way interactions equally
easily.
 However, one-way interactions cannot be directly
programmed in the message-passing paradigm because the
source of the data to be transferred must explicitly send
the data to the recipient.

Dr. Hanif Durad 142
 In the message-passing paradigm, all one-way
interactions must be converted to two-way interactions
via program restructuring
 Static one-way interactions can be easily converted to
two-way communications. Since the time and the
location in the program of a static one-way interaction is
known a priori, introducing a matching interaction
operation in the partner task is enough to convert a one-
way static interaction to a two-way static interaction.

Dr. Hanif Durad 143
 On the other hand, dynamic one-way interactions can
require some nontrivial program restructuring to be
converted to two-way interactions.
 The most common such restructuring involves polling.
Each task checks for pending requests from other tasks
after regular intervals, and services such requests, if
any.

Tasks vs. Processes vs.
Processors
 Processors
 are the hardware units that physically perform
computations.
 in most cases, there is a one-to-one correspondence
between processes and processors
 The tasks, into which a problem is decomposed,
run on physical processors.

Tasks vs. Processes vs.
Processors (2/2)
 A process
 refers to a processing or computing agent that performs
tasks.
 is an abstract entity that uses the code and data
corresponding to a task to produce the output of that task
within a finite amount of time after the task is activated by
the parallel program.
 In addition to performing computations, a process may
synchronize or communicate with other processes, if
needed.
 In order to obtain any speedup over a sequential
implementation, a parallel program must have several
processes active simultaneously, working on different tasks

3.4 Mapping Techniques-A Good
Mapping (1/6)
 Mapping = mechanism by which tasks are assigned to
processes for execution
 The task-dependency and task-interaction graphs that
result from a choice of decomposition play an important
role in the selection of a good mapping
 A good mapping should seek to
1. Max.the use of concurrency by mapping independent tasks
onto different processes,
2. Min. the total completion time by ensuring that processes are
available to execute the tasks on the critical path as soon as
such tasks become executable
3. Min. interaction among processes by mapping tasks with a
high degree of mutual interaction onto the same process.

A Good Mapping (2/6)
 In most nontrivial parallel algorithms, these three tend to
be conflicting goals.
 Example: the most efficient decomposition-mapping
combination is a single task mapped onto a single process. It
wastes no time in idling or interacting, but achieves no speedup
either.
 Mapping of tasks onto processes plays an important role
in determining how efficient the resulting parallel
algorithm is.
 Even though the degree of concurrency is determined by the
decomposition, it is the mapping that determine how much of
that concurrency is actually utilized & how efficiently

 Once a computation has been decomposed into
tasks, these tasks are mapped onto processes with
the object that all tasks complete in the shortest
amount of elapsed time.
 To achieve a small execution time, overheads of
executing the tasks in parallel must be minim.
 For a given decomposition, there are two key
sources of overhead.
1. The time spent in inter-process interaction.
2. The time that some processes may spend being idle.

 Processes can be idle even before the overall
computation is finished for a variety of reasons:
 Uneven load distribution may cause some processes
to finish earlier
 All the unfinished tasks mapped onto a process may
be waiting for tasks mapped onto other processes to
finish in order to satisfy the constraints imposed by
the task-dependency graph.

 Both interaction and idling are often a function
of mapping.
 A good mapping of tasks onto processes must
strive to achieve the objective of reducing
1. the amount of time processes spend in interacting
with each other,
2. the total amount of time some processes are idle
while the others are performing some tasks.

 These two objectives often conflict with each other.
 Example:
 Min. the interactions can be easily achieved by assigning sets
of tasks that need to interact with each other onto the same
process.
 Such a mapping will result in a highly unbalanced workload
among the processes.
 Following this strategy to the limit will often map all tasks
onto a single process.
 Processes with a lighter load will be idle when those with a
heavier load are finishing their tasks.
 To balance the load among processes => assign tasks that
interact heavily to different processes.

Example: Two mappings (1/3)
 A task-dependency graph determines which tasks can
execute in parallel and which must wait for some others to
finish at a given stage in the execution of a parallel
algorithm
 Poor synchronization among interacting tasks can lead to
idling if one of the tasks has to wait to send or receive
data from another task.
 A good mapping ensure that the computations and
interactions among processes at each stage of the
execution of the parallel algorithm are well balanced.

Example: Two mappings (2/3)
 Fig. shows two mappings of 12-task
decomposition in which the last 4 tasks can be
started only after the first 8 are finished due to
dependencies among tasks.
 As the figure shows, two mappings, each with an
overall balanced workload, can result in different
completion times.

Example: Two mappings of a hypothetical
decomposition with a synchronization
Matrix Multiplication-Intermediate Data Partitioning: Example
(3/3)
Synchronization (?)

3.4 Mapping classification
 The choice of a good mapping in this case depends upon
several factors, including
 the knowledge of task sizes,
 the size of data associated with tasks,
 the characteristics of inter-task interactions, and
 the parallel programming paradigm.
 Obtaining an optimal mapping is an NP-complete
problem for non-uniform tasks.
 Two types of Mapping:
 Static and Dynamic

3.4.A Static Mapping
 Distribute the tasks among processes prior to the execution of the
algorithm.
 For statically generated tasks, either static or dynamic mapping can
be used.
 For many practical cases, relatively inexpensive heuristics provide
fairly acceptable approximate solutions to the optimal static
mapping problem.
 Algorithms that make use of static mapping are in general easier to
design and program.
 If task sizes are unknown, a static mapping can potentially lead to
serious load-imbalances
Dr. Hanif Durad 156

3.4.B Dynamic Mapping
 Distribute the work among processes during the execution of the
algorithm.
 If tasks are generated dynamically, then they must be mapped
dynamically too.
 If the amount of data associated with tasks is large relative to the
computation, then a dynamic mapping may entail moving this data
among processes.
 In a shared-address-space paradigm, dynamic mapping may work
well even with large data associated with tasks if the interaction is
read-only.
 Algorithm that require dynamic mapping are complicated,
particularly in the message passing programming paradigm

3.4.1 Schemes for Static
Mapping:
1. Static Mappings Based on Data Partitioning
1. Array or Metrics Distribution Schemes
1. Block Distribution
2. Block-Cyclic Distributions
3. Cyclic Distributions
4. Randomized Block Distributions
2. Graph partitioning
2. Static Mappings Based on Task Partitioning
3. Hierarchical Static Mappings

3.4.1.1 Static Mappings Based on
Data Partitioning
 is often-used in conjunction with a decomposition based on data
partitioning.
 used for mapping certain problems that are expressed naturally by a
static task-dependency graph
 In a decomposition based on partitioning data, the tasks are closely
associated with portions of data by the owner-computes rule.
 ⇒ mapping the relevant data onto the processes is equivalent to
mapping tasks onto processes.
 Two of the most common ways of representing data in algorithms:
 arrays and
 graphs.

3.4.1.1.1 Array or Metrics
Distribution Schemes
1. Block Distribution
2. Block-Cyclic Distributions
3. Cyclic Distributions
4. Randomized Block Distributions

3.4.1.1.1.1 Block Distribution
Schemes
 Simplest ways to distribute an array and assign uniform
contiguous portions of the array to different processes.
 In these distributions, a d-dimensional array is
distributed among the processes such that each process
receives a contiguous block of array entries along a
specified subset of array dimensions.
 Block distributions of arrays are particularly suitable
when there is a locality of interaction, i.e., computation
of an element of an array requires other nearby
elements in the array.

Example 1: One-Dimensional Block
Partitioning of an Array (1/2)
 Row-wise array distributions:
 Partition the array into p parts such that the kth part
contains rows kn/p...(k + 1)n/p - 1, where 0 <=k < p.
 Each partition contains a block of n/p consecutive
rows
 Column-wise array distributions:
 Partition A along the second dimension, then each
partition contains a block of n/p consecutive columns.

Example 1: One-dimensional Block
Partitioning of an Array Among
Eight Processes (2/2)
Dr. Hanif Durad 163

Example 2: Two-Dimensional Block
Distributions of an Array (1/2)
 Instead of selecting a single dim, select multiple
dims
 Select both dimensions and partition the matrix
into blocks such that each block corresponds to a
n/p1 x n/p2 section of the matrix, with p = p1 x p2
being the number of processes.
Dr. Hanif Durad 164

Example 2: Two-dimensional Block
Distributions of an Array (2/2)
Dr. Hanif Durad 165
(a) on a 4 x 4 process grid,
(b) on a 2 x 8 process grid

Example 3: Matrix Multiplication-
Block Distribution (1/5)
 Consider the n x n matrix multiplication C = A x B.
 Partition the output matrix C.
 Balance the computations by partitioning C uniformly
among the p available processes.
1. 1-D block distribution: each process will get a block of
n/p rows (or columns) of C,
2. 2-D block distribution: each process will get a block of
size
Dr. Hanif Durad 166
( ) x ( )n p n p

Matrix Multiplication- Block
Distribution (2/5)
 Higher dim. distributions allow to use more processes.
1. 1-D block distribution/ matrix-matrix multiplication: up
to n processes by assigning a single row of C to each
process.
2. 2-D distribution will allow us to use up to n2
processes by
assigning a single element of C to each process.
3. Higher dim. distributions reduce the amount of
interactions among the different processes for many
problems.

Matrix Multiplication- Block
Distribution (3/5)
 Higher dim. distributions reduce the amount of interactions
among the different processes for many problems
1. 1-D partitioning along the rows, each process needs to
access the corresponding ‘n/p’ rows of matrix A and the
entire matrix B: the total amount of data that needs to be
accessed is ‘n2
/p + n2’
.
2. 2-D distribution: each process needs to access rows of
matrix A and columns of matrix B: the total amount
of shared data that each process needs to access is
in 1-d case
( )n p
( )n p
2 2
( / ) ( / )n p n po o<< 2 2 2
Total / / 2 /n p n p n p= + =

Dr. Hanif Durad 169
Data sharing needed for matrix multiplication with
(a) one-dimensional partitioning of the output matrix.
Shaded portions of the input matrices A and B are
required by the process that computes the shaded
portion of the output matrix C
n x n n x n n x n
(4/5)

Dr. Hanif Durad 170
Data sharing needed for matrix multiplication with
(b) two dimensional partitioning of the output
matrix.
Shaded portions of the input matrices A and B are
required by the process that computes the shaded
portion of the output matrix C
(5/5)

Example 4: Block LU
Factorization (1/7)
 If the amount of work differs for different elements of a matrix, a
block distribution can potentially lead to load imbalances.
 Classic example of this phenomenon: LU factorization of a matrix:
the amount of computation increases from the top left to the bottom
right of the matrix.
 LU factorization alg. factors a nonsingular square matrix A into the
product of
 a lower triangular matrix L with a unit diagonal and
 an upper triangular matrix U.
 The following alg. shows the serial column-based algorithm.
 Matrices L and U share space with A: A is modified to store L and U in it

Dr. Hanif Durad 172
A serial Column-based LU
Factorization Algo. (2/7)

Block LU Factorization (3/7)
 Fig. shows a possible decomposition of LU
factorization into 14 tasks using a 3 x 3 block
partitioning of the matrix and using a block version
 For each iteration of the outer loop k := 1 to n, the next
nested loop in the above algorithm goes from k + 1 to n.
 The active part of the matrix, as shown in Fig. 2,
shrinks towards the bottom right corner of the matrix as
the computation proceeds.
 In a block distribution, the processes assigned to the
beginning rows & columns would perform far less
work than those assigned to the later rows & cols.

A decomposition of LU factorization into 14 tasks -
notice the significant load imbalance.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:

Dr. Hanif Durad 175
Block-Cyclic Distribution for LU
Decomposition (5/7)

 Computing different blocks of the matrix requires different
amounts of work
 Illustrated in Fig.
 Comp. value of A1,1 requires only one task – Task 1.
 Comp. value of A3,3 requires 3 tasks –9, 13, and 14.
 The process working on a block may idle even when there
are unfinished tasks associated with that block.
 This idling can occur if the constraints imposed by the task-
dependency graph do not allow the remaining tasks on this
process to proceed until one or more tasks mapped onto other
processes are completed.

Block-Cyclic Distribution for LU
Decomposition (7/7)
One- and two-dimensional block-cyclic
distributions among 4 processes.

3.4.1.1.1.2 Block-Cyclic
Distribution
 The block-cyclic distribution is a variation of the
block distribution scheme that can be used to
alleviate the load-imbalance and idling problems.
 Central idea behind a block-cyclic distribution:
 partition an array into many more blocks than the
number of available processes.
 assign the partitions (and the associated tasks) to
processes in a round-robin manner so that each
process gets several non-adjacent blocks.

Example 1: 1-D Block-Cyclic
Distribution of a Matrix (1/2)
 Example 1: a 1-d block-cyclic distribution of a matrix
among p processes,
 the rows (columns) of an n x n matrix are divided into αp
groups of n/(αp) consecutive rows (columns), where
1 <=α<=n/p.
 These blocks are distributed among the p processes in a
wraparound fashion such that block bi is assigned to process
Pi %p ('%' is the modulo operator).
 Assigns α blocks of the matrix to each process, but each
subsequent block that gets assigned to the same process is p
blocks away.

1-D Block-Cyclic Distribution (2/2)
Example of 1-D block-cyclic distributions among four processes.
The rows of the array are grouped into blocks each consisting of two
rows, resulting in eight blocks of rows. These blocks are distributed to
four processes in a wraparound fashion.
α=2;
n=16;
p=4
Groups 8pα= =
Rows (Columns) in each
Group /( ) 2n pα= =

Example 2: 2-D Block-Cyclic
Distribution of a Matrix (1/2)
 Example 2: a 2-d block-cyclic distribution of an n
x n array by partitioning it into
 square blocks of size and distributing them
on a hypothetical array of processes in a
wraparound fashion.
 The block-cyclic distribution can be extended to arrays
of higher dimensions.
xp pα α
xp p

Dr. Hanif Durad 182
2-D Block- Cyclic Distribution (2/2)
Example : two-dimensional block-cyclic distributions among four
processes. The matrix is blocked into 16 blocks each of size 4 × 4, and
it is mapped onto a 2 × 2 grid of processes in a wraparound fashion.
α=2;
n=16;
p=4
Blocksize x 4x4p pα α= =
Processor Array x 2x2p p= =

3.4.1.1.1.3 Cyclic Distribution
(1/2)
 Note that if we increase α to its upper limit of n/p,
 then each block is a single row (column) of the matrix in a
one-dimensional block-cyclic distribution and
 a single element of the matrix in a two-dimensional block-
cyclic distribution.
 Such a distribution is known as a cyclic distribution.
 Cyclic distribution = extreme block-cyclic distribution
 Can result in an almost perfect load balance due to the
extreme fine-grained decomposition.

Cyclic Distribution (2/2)
 Since a process does not have any contiguous data to
work on, the resulting lack of locality may result in
serious performance penalties.
 Additionally, such a decomposition usually leads to a
high degree of interaction relative to the amount
computation in each task.
 The lower limit of 1 for the value of a results in
maximum locality and interaction optimality, but the
distribution degenerates to a block distribution.
Therefore, an appropriate value of a must be used to
strike a balance between interaction conservation and
load balance.

Block- Cyclic Distribution-Problems
(1/2)
 In case of sparse matrices, block decompositions are
more complex.
 Consider the problem of multiplying a sparse matrix
with a vector.
 The graph of the matrix is a useful indicator of the
work (number of nodes) and communication (the
degree of each node).
 In this case, we would like to partition the graph so as
to assign equal number of nodes to each process, while
minimizing edge count of the graph partition

Dr. Hanif Durad 186
Block- Cyclic Distribution Problem
(2/2)
Using the block-cyclic distribution shown in (b) to distribute the
computations performed in array (a) will lead to load imbalances.

3.4.1.1.1.4 Randomized Block
Distributions (1/3)
 Load balance is sought by partitioning the array
into many more blocks than the number of
available processes.
 The blocks are uniformly and randomly
distributed among the processes

Dr. Hanif Durad 188
Randomized Block Distributions (2/3)
Example-1
A one-dimensional randomized block mapping of 12 blocks onto four
process (i.e., α = 3).

Dr. Hanif Durad 189
Randomized Block Distributions (3/3)
Example-2
Using a two-dimensional random block distribution shown in (b) to
distribute the computations performed in array (a), as shown in (c).

3.4.1.1.2 Graph Partitioning (1/3)
 there are many algorithms that operate on sparse
data structures and for which the pattern of
interaction among data elements is data
dependent and highly irregular.
 Numerical simulations of physical phenomena provide
a large source of such type of computation

In these computations, the physical domain is discretized
and represented by a mesh of elements.

Ex: Simulation of a physical phenomenon such the
dispersion of a water contaminant in the lake involve
computing the level of contamination at each vertex of this
mesh at various intervals of time.

Graph Partitioning (2/3)
 Random: each process will need to access a large set of
points belonging to other processes to complete
computations for its assigned portion of the mesh.
 ? Partition the mesh into p parts such that each part
contains roughly the same no. of mesh-points or vertices,
& no. of edges that cross partition boundaries is
minimized
 NP-complete problem.
 Algorithms that employ powerful heuristics are available to
compute reasonable partitions.
 Each process is assigned a contiguous region of the mesh such
that the total number of mesh points that needs to be accessed
across partition boundaries is minimized.

Dr. Hanif Durad 192
Partitioning the Graph of Lake
Superior (3/3)
Random Partitioning
Partitioning for
minimum edge-cut

3.4.1.2 Static mapping: Mappings
Based on Task Partitioning
 When the computation is naturally expressible in the form of a
static task-dependency graph with tasks of known sizes
 Conflicting objectives of minimizing idle time and minimizing the
interaction time of the parallel algorithm
 Determining an optimal mapping for a general task-dependency
graph is an NP-complete problem
 Ex:
 task-dependency graph that is a perfect binary tree (e.g. finding
minimum)
 mapping on a hypercube
 minimizes the interaction overhead by mapping many
interdependent tasks onto the same process (i.e., the tasks along a
straight branch of the tree) and others on processes only one
communication link away from each other.

Example 1:Static mapping: Based on Task
Partitioning for a Binary Tree Dependency
Graph

Task Partitioning for Computing a
Sparse Matrix-Vector Product (1/3)
 An approximate solution
 Ex: A mapping of three for sparse matrix-vector multiplication
onto three processes
 First: this mapping assigns tasks corresponding to four
consecutive entries of b to each process.
 Second: partitioning for the task interaction graph
 Ci contains the indices of b that the tasks on Process i need to
access from tasks mapped onto other processes
 A quick comparison of the lists C0, C1, and C2 in the two cases
readily reveals that the second mapping entails fewer exchanges
of elements of b between processes than the first mapping.

Example 2: Task Partitioning for
Computing a Sparse Matrix-Vector Product
(2/3)

Task Partitioning for Computing
Sparse Matrix-Vector Product
(3/3)

3.4.1.3 Static Mapping: Hierarchical
Mappings
 Sometimes a single mapping technique is inadequate
 A mapping based solely on the task dependency graph
may suffer from load-imbalance or inadequate
concurrency
 If the tasks are large enough, then a better mapping can
be obtained by a further decomposition of the tasks into
smaller subtasks.
 For example, the task mapping of the binary tree
(quicksort) cannot use a large number of processors.
 For this reason, task mapping can be used at the top
level and data partitioning within each level.

An example of task partitioning at top level with
data partitioning at the lower level.
Example: Hierarchical Mappings

3.4.2 Schemes for Dynamic
Mapping
 necessary in situations
 where a static mapping may result in a highly
imbalanced distribution of work among processes or
 where the task-dependency graph itself if dynamic,
thus precluding a static mapping.
 the primary reason for using a dynamic mapping
is balancing the workload among processes,
 dynamic mapping is often referred to as
dynamic
 centralized or
 distributed

3.4.2.1 Dynamic Mapping:
Centralized Schemes (1/3)
 all executable tasks are maintained in a common central data
structure or they are maintained by a special process or a subset of
processes.
 a special process is designated to manage the pool of available
tasks, then it is often referred to as the master
 the other processes that depend on the master to obtain work are
referred to as slaves
 whenever a process has no work, it takes a portion of available
work from the central data structure or the master process.
 whenever a new task is generated, it is added to this centralized
data structure or reported to the master process.
 centralized load-balancing schemes are usually easier to implement
than distributed schemes, but may have limited scalability.

Dynamic Mapping: Centralized
Schemes (2/3)
 the large no. accesses to the common data structure or the master
process tends to become a bottleneck.
 Example:
 the problem of sorting the entries in each row of an n x n matrix A.
 A naive mapping - an equal no. rows to each process - may lead to
load-imbalance.
 Another solution: maintain a central pool of indices of the rows
that have yet to be sorted.
 Whenever a process is idle, it picks up an available index, deletes it, and
sorts the row with that index
 Scheduling the independent iterations of a loop among parallel processes is
known as self scheduling

Dynamic Mapping: Centralized
Schemes (3/3)
 When the number of processes increases, the master may
become the bottleneck
 To alleviate this, a process may pick up a number of
tasks (a chunk) at one time. This is called Chunk
scheduling.
 Selecting large chunk sizes may lead to significant load
imbalances as well.
 A number of schemes have been used to gradually
decrease chunk size as the computation progresses.

3.4.2.2 Dynamic Mapping:
Distributed Schemes
 the set of executable tasks are distributed among
processes which exchange tasks at run time to balance
work
 each process can send work to or receive work from any
other process
 the critical parameters of a distributed load balancing
scheme are as follows:
 How are the sending and receiving processes paired together?
 Is the work transfer initiated by the sender or the receiver?
 How much work is transferred in each exchange?
 When is the work transfer performed?

3.5 Methods for Containing
Interaction Overheads
 Reducing the interaction overhead among concurrent tasks is
important for an efficient parallel program.
 The overhead that a parallel program incurs due to interaction
among its processes depends on many factors:
 the volume of data exchanged during interactions,
 the frequency of interaction,
 the spatial and temporal pattern of interactions, etc.
 Some general techniques that can be used to reduce the
interaction overheads incurred by parallel programs:
1. Maximizing Data Locality and Minimizing Contention and Hot Spots
2. Overlapping Computations with Interactions
3. Replicating Data or Computations
4. Using Optimized Collective Interaction Operations
5. Overlapping Interactions with Other Interactions

3.5.1 Maximizing Data Locality
(1/3)
 Data locality enhancing techniques encompass a
wide range of schemes that try to
 minimize the volume of non-local data that are
accessed,
 maximize the reuse of recently accessed data, and
 minimize the frequency of accesses.

Maximizing Data Locality (2/3)
 3.5.1.1 Minimize Volume of Data-Exchange
 by using appropriate decomposition and mapping schemes.
 Example: matrix multiplication

using a two-dimensional mapping of the computations to the processes we
reduce the amount of shared data that needs to be accessed by each task as
opposed to a one dimensional mapping
 Example: Dot product of two vectors

two vectors of length n in parallel such that each of the p tasks multiplies
n/p pairs of elements.

Don't add individual product of a pair of numbers to the final result

create a partial dot product of its assigned portion of the vectors of length
n/p in its own local location,

only access the final shared location once to add this partial result.

This will reduce the number of accesses
2 2 2
2 / /n p n p n+=

Maximizing Data Locality (3/3)
 3.5.1.2 Minimize Frequency of Interactions
 by restructuring the algorithm such that shared data
are accessed and used in large pieces

3.5.2 Minimizing Contention and
Hot Spots
 contention occurs when
 multiple tasks try to access the same resources
concurrently.
 multiple simultaneous transmissions of data over the
same interconnection link,
 multiple simultaneous accesses to the same memory
block,
 multiple processes sending messages to the same
process at the same time

3.5.3 Overlapping Computations
with Interactions
 There are a no. techniques that can be used
 The simplest: initiate an interaction early enough so that it is
completed before it is needed for computation.
 need to identify computations that can be performed before the interaction and
do not depend on it
 Then the parallel program must be structured to initiate the interaction at an
earlier point in the execution than it is needed in the original algorithm
 In certain dynamic mapping schemes, as soon as a process runs out
of work, it requests and gets additional work from another process.
 If the process can anticipate that it is going to run out of work and initiate a
work transfer interaction in advance
 On a shared-address-space arch, assisted by prefetching hardware.
 The prefetch hardware can anticipate the memory addresses that will need to
be accessed in the immediate future, and can initiate the access in advance of
when they are needed

3.5.4 Replicating Data or
Computations
 Replicating Data
 Replicating Computations

3.5.4.1 Replicating Data
 Example:
 multiple processes may require frequent read-only access to
shared data structure, such as a hash-table, in an irregular pattern.
 replicate a copy of the shared data structure on each process
 after the initial interaction during replication, all subsequent
accesses to this data structure are free of any interaction overhead
 Increases the memory requirements of a parallel program
 The aggregate amount of memory required to store the replicated
data increases linearly with the no. concurrent processes.
 This may limit the size of the problem that can be solved on a
given parallel computer.

3.5.4.2 Computation Replication
 the processes in a parallel program often share intermediate
results.
 In some situations, it may be more cost-effective for a process to compute
these intermediate results than to get them from another process that
generates them.
 Example:
 Fast Fourier Transform, on an N-point series, N distinct powers of ω or
"twiddle factors" are computed and used at various points in the
computation.
 In a parallel implementation of FFT, different processes require
overlapping subsets of these N twiddle factors.
 Message-passing paradigm: each process locally compute all the twiddle
factors it needs.

Although the parallel algorithm may perform many more twiddle factor
computations than the serial algorithm, it may still be faster than sharing the
twiddle factors.

3.5.5 Using Optimized Collective
Interaction Operations (1/2)
 Often the interaction patterns among concurrent activities are
static and regular.
 A class of such static and regular interaction patterns are those
that are performed by groups of tasks, and they are used to
achieve regular data accesses or to perform certain type of
computations on distributed data.
 A number of key such collective interaction operations have
been identified that appear frequently in many parallel
algorithms.
 Examples:
 Broadcasting some data to all the processes or
 adding up numbers, each belonging to a different process

Using Optimized Collective
Interaction Operations (2/2)
 The collective data-sharing operations can be classified into three
categories.
1. operations that are used by the tasks to access data,
2. operations used to perform some communication-intensive
computations
3. used for synchronization.
 Highly optimized implementations of these collective operations
have been developed that minimize the overheads due to data
transfer as well as contention.
 Optimized implementations of these collective interaction
operations are available in library form from the vendors of most
parallel computers, e.g., MPI (message passing interface).
 the algorithm designer does not need to think about how these
operations are implemented and needs to focus only on the
functionality achieved by these operations.

3.5.6 Overlapping Interactions
with Other Interactions
 Overlapping interactions between multiple pairs of processes can
reduce the effective volume of communication.
 Example:
 the commonly used collective communication operation of one-to-all
broadcast in a message-passing paradigm with four processes P0, P1, P2, and
P3.
 A commonly used algorithm to broadcast some data from P0 to all
other processes works as follows.
 In the first step, P0 sends the data to P2.
 In the second step, P0 sends the data to P1, and concurrently, P2 sends the
same data that it had received from P0 to P3.
 The entire operation is thus complete in two steps because the two interactions
of the second step require only one time step.
 This operation is illustrated in Figure (a).
 A naive broadcast algorithm would send the data from P0 to P1 to
P2 to P3, thereby consuming three steps as illustrated in Figure (b).

One-to-all Broadcast of One Message
Commonly Used
Algorithm
Naive
Algorithm
Illustration of overlapping interactions in broadcasting
data from one to four processes

One-to-all Broadcast of 4 Messages
using Naive Algorithm
Illustration of overlapping interactions in broadcasting data
from one to four processes using Naive Algorithm
Total steps=6
Using first algo.=?=8

3.6 Parallel Algorithm Models
 An algorithm model is typically a way of structuring a
parallel algo.
 by selecting a decomposition and mapping technique and
 by applying the appropriate strategy to minimize interactions
 Models
 data parallel,
 task graph,
 work pool,
 master-slave
 pipeline
 hybrids

3.6.1 Data Parallel Model (1/2)
 The data-parallel model is one of the simplest algorithm models.
 the tasks are statically or semi-statically mapped onto processes and each task
performs similar operations on different data.
 This type of parallelism that is a result of identical operations being applied
concurrently on different data items is called data parallelism.
 The work may be done in phases and the data operated upon in different
phases may be different.
 Typically, data-parallel computation phases are interspersed with interactions
to synchronize the tasks or to get fresh data to the tasks.
 Since all tasks perform similar computations, the decomposition of the
problem into tasks is usually based on data partitioning
 because a uniform partitioning of data followed by a static mapping is
sufficient to guarantee load balance

Data Parallel Model (2/2)
 Data-parallel algorithms can be implemented in both shared-address
space and message-passing paradigms.
 The partitioned address-space in a message-passing paradigm may allow better
control of placement, and thus may offer a better handle on locality.
 Shared-address space can ease the programming effort, especially if the
distribution of data is different in different phases of the algorithm.
 Interaction overheads in the data-parallel model can be minimized
 by choosing a locality preserving decomposition
 by overlapping computation and interaction and
 by using optimized collective interaction routines.
 A key characteristic is that for most problems, the degree of data
parallelism increases with the size of the problem, making it possible
to use more processes to effectively solve larger problems.
 An example of a data-parallel algorithm is dense matrix multiplication
SIMD (?)

3.6.2 Task Graph Model (1/2)
 The task-dependency graph may be either trivial, as in the case of
matrix multiplication, or nontrivial.
 In certain parallel algorithms, the task-dependency graph is explicitly used
in mapping
 In the task graph model, the interrelationships among the tasks are
utilized to promote locality or to reduce interaction costs.
 Employed to solve problems in which the amount of data
associated with the tasks is large relative to the amount of
computation associated with them
 Work is more easily shared in paradigms with globally addressable
space, but mechanisms are available to share work in disjoint
address space.

Task Graph Model (2/2)
 Typical interaction-reducing techniques applicable to this model
include
 reducing the volume and frequency of interaction by promoting locality
while mapping the tasks based on the interaction pattern of tasks, and
 using asynchronous interaction methods to overlap the interaction with
computation.
 Examples of algorithms based on the task graph model include
 parallel quicksort,
 sparse matrix factorization,
 parallel algorithms derived via divide-and-conquer decomposition.
 This type of parallelism that is naturally expressed by independent
tasks in a task-dependency graph is called task parallelism.

3.6.3 Work Pool Model (Task
Pool Model) (1/3)
 Characterized by a dynamic mapping of tasks onto processes for load
balancing, any task may potentially be performed by any process.
 There is no desired pre-mapping of tasks onto processes.
 The mapping may be centralized or decentralized.
 Pointers to the tasks may be stored in a physically shared list, priority
queue, hash table, or tree, or they could be stored in a physically
distributed data structure
 Work may be statically available in the beginning, or could be
dynamically generated;
 i.e., the processes may generate work and add it to the global (possibly
distributed) work pool.

Work Pool Model (2/3)
 If work is generated dynamically and a decentralized
mapping is used,
 then a termination detection algo. would be required so that all
processes can actually detect the completion of the entire
program and stop looking for more work.
 In the message-passing paradigm, the work pool model is
typically used when the amount of data associated with
tasks is relatively small compared to the computation
associated with the tasks.
 tasks can be readily moved around without causing too much
data interaction overhead.

Work Pool Model (3/3)
 Granularity of the tasks can be adjusted to attain the
desired level of tradeoff between load-imbalance &
overhead of accessing the work pool for adding and
extracting tasks.
 Parallelization of loops by chunk scheduling or related
methods is an example of the use of the work pool model
with centralized mapping when the tasks are statically
available.
 Ex. work pool model where the tasks are generated
dynamically:
 Parallel tree search where the work is represented by a
centralized or distributed data structure

3.6.4 Master-Slave Model
(Manager-Worker) (1/4)
 One or more master processes generate work and
allocate it to worker processes.
 the tasks may be allocated a priori if the manager can
estimate the size of the tasks or
 if a random mapping can do an adequate job of load
balancing.
 In another scenario, workers are assigned smaller pieces
of work at different times.

preferred if it is time consuming for the master to generate
work and hence it is not desirable to make all workers wait
until the master has generated all work pieces.

Master-Slave Model (2/4)
 In some cases, work may need to be performed in
phases, and work in each phase must finish before
work in the next phases can be generated.
 In this case, the manager may cause all workers to
synchronize after each phase.
 Usually: no desired pre-mapping of work to
processes; any worker can do any job assigned.

 The model can be generalized to hierarchical or multi-level
manager-worker model
 the top-level manager feeds large chunks of tasks to second-level
managers, who further subdivide the tasks among their own
workers and may perform part of the work themselves.
 This model is generally equally suitable to shared-address-
space or message-passing paradigms since the interaction
is naturally two-way;
 the manager knows that it needs to give out work and
 workers know that they need to get work from the manager.

 Care should be taken
 to ensure that the master does not become a bottleneck
(may happen if tasks are too small/fast)
 granularity of tasks: cost of doing work >> cost of
transferring work & cost of synchronization.
 asynchronous interaction may help

overlap interaction and the computation associated with work
generation by the master.

It may also reduce waiting times if the nature of requests from
workers is nondeterministic.

3.6.5 Pipeline or Producer-
Consumer Model (1/3)
 A stream of data is passed on through a succession of
processes, each of which perform some task on it.
 This simultaneous execution of diff. programs on a data
stream is called stream parallelism.
 With the exception of the process initiating the pipeline,
the arrival of new data triggers the execution of a new
task by a process in the pipeline.
 The processes could form such pipelines in the shape of
linear or multidimensional arrays, trees, or general
graphs with or without cycles.

Pipeline or Producer-Consumer
Model (2/3)
 A pipeline is a chain of producers and consumers.
 Each process in the pipeline can be viewed as

a consumer of a sequence of data items for the process preceding it in the pipeline
and

as a producer of data for the process following it in the pipeline.
 The pipeline does not need to be a linear chain; it can be a directed
graph.
 The pipeline model usually involves a static mapping of tasks onto
processes.
 Load balancing is a function of task granularity.
 The larger the granularity, the longer it takes to fill up the pipeline,
 for the trigger produced by the first process in the chain to propagate to the
last process, thereby keeping some of the processes waiting.

Pipeline or Producer-Consumer
Model (3/3)
 Too fine a granularity may increase interaction overheads
because processes will need to interact to receive fresh
data after smaller pieces of computation.
 The most common interaction reduction technique
applicable to this model is overlapping interaction with
computation.
 An example of a two-dimensional pipeline is the parallel
LU factorization algorithm
Problem 3.5

3.6.5 Hybrid models (1/2)
 In some cases, more than one model may be
applicable to the problem at hand, resulting in a
hybrid algorithm model.
 A hybrid model may be composed
 either of multiple models applied hierarchically
 or multiple models applied sequentially to different
phases of a parallel algorithm.

Hybrid models (2/2)
 In some cases, an algorithm formulation may have
characteristics of more than one algorithm model.
 For instance, data may flow in a pipelined manner in a pattern
guided by a task-dependency graph.
 In another scenario, the major computation may be described
by a task dependency graph, but each node of the graph may
represent a supertask comprising multiple subtasks that may be
suitable for data parallel or pipelined parallelism.
 Parallel quicksort is one of the applications for which a
hybrid model is ideally suited.

Chapter 3 pc

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Chapter 3 pc (20)

More from Hanif Durad (14)

Recently uploaded (20)

Chapter 3 pc