Parallel Processing

Parallel Processing

Dr. Pierre Vignéras
http://www.vigneras.name/pierre

This work is licensed under a Creative Commons
Attribution-Share Alike 2.0 France.
See
http://creativecommons.org/licenses/by-sa/2.0/fr/
for details


Text & Reference Books

● Fundamentals of Parallel Processing
Harry F. Jordan & Gita Alaghband
Pearson Education
ISBN: 81-7808-992-0
Parallel & Distributed Computing Handbook
●

Albert Y. H. Zomaya
McGraw-Hill Series on Computer Engineering
ISBN: 0-07-073020-2

Parallel Programming (2nd edition)
●
Books

Barry Wilkinson and Michael Allen
Pearson/Prentice Hall
ISBN: 0-13-140563-2

Class, Quiz & Exam Rules
● No entry after the first 5 minutes
● No exit before the end of the class
● Announced Quiz
– At the beginning of a class
– Fixed timing (you may suffer if you arrive late)
– Spread Out (do it quickly to save your time)
– Papers that are not strictly in front of you will be
considered as done
– Cheaters will get '1' mark
Rules

– Your head should be in front of your paper!


Grading Policy
● (Temptative, can be slightly modified)
– Quiz: 5 %
– Assignments: 5%
– Project: 15%
– Mid: 25 %
Grading Policy

– Final: 50 %


Outline
(Important points)
● Basics
– [SI|MI][SD|MD],
– Interconnection Networks
– Language Specific Constructions
● Parallel Complexity
– Size and Depth of an algorithm
– Speedup & Efficiency
● SIMD Programming
– Matrix Addition & Multiplication
– Gauss Elimination


Basics

●
Flynn's Classification
●
Interconnection Networks
Outline

●
Language Specific Constructions

Dr. Pierre Vignéras 6

History
● Since
– Computers are based on the Von Neumann
Architecture
– Languages are based on either Turing Machines
(imperative languages), Lambda Calculus
(functional languages) or Boolean Algebra (Logic
Programming)
● Sequential processing is the way both
I. Basics

computers and languages are designed
● But doing several things at once is an obvious
way to increase speed


Definitions
● Parallel Processing objective is execution speed
improvement
● The manner is by doing several things at once
● The instruments are
– Architecture,
– Languages,
– Algorithm
I. Basics

● Parallel Processing is an old field
– but always limited by the wide sequential world for
various reasons (financial, psychological, ...)


Flynn Architecture
Characterization
● Instruction Stream (what to do)
– S: Single, M: Multiple
● Data Stream (on what)
– I: Instruction, D: Data
● Four possibilities
– SISD: usual computer
– SIMD: “Vector Computers” (Cray)
I. Basics

– MIMD: MultiProcessor Computers
– MISD: Not very useful (sometimes presented as
pipelined SIMD)


SISD Architecture
● Traditional Sequential Computer
● Parallelism at the Architecture Level:
– Interruption
– IO processor
– Pipeline

Flow of Instructions I8 I7 I6 I5
I. Basics

IF OF EX STR I4 I3 I2 I1

4-stages pipeline


Problem with pipelines
● Conditional branches need complete flush
– The next instruction depends on the final result of
a condition
● Dependencies
– Wait (inserting NOP in pipeline stages) until a
previous result becomes available
● Solutions:
– Branch prediction, outoforder execution, ...
I. Basics

● Consequences:
– Circuitry Complexities (cost)
– Performance

Exercises: Real Life Examples
● Intel:
● Pentium 4:
● Pentium 3 (=M=Core Duo):
● AMD
● Athlon:
● Opteron:
● IBM
● Power 6:
I. Basics

● Cell:
● Sun
● UltraSparc:
● T1:

Instruction Level Parallelism
● Multiple Computing Unit that can be used
simultaneously (scalar)
– Floating Point Unit
– Arithmetic and Logic Unit
● Reordering of instructions
– By the compiler
– By the hardware
I. Basics

● Lookahead (prefetching) & Scoreboarding
(resolve conflicts)
– Original Semantic is Guaranteed


Reordering: Example
● EXP=A+B+C+(DxExF)+G+H
+ +

+ H + H
Height=5

Height=4
+ G + x
+ x + + x F

+ C x F A B C G D E
I. Basics

A B D E
4 Steps required if
5 Steps required if
hardware can do 2 adds and
hardware can do 1 add and
1 multiplication simultaneously
1 multiplication simultaneously

SIMD Architecture
● Often (especially in scientific applications),
one instruction is applied on different data
for (int i = 0; i < n; i++) {
r[i]=a[i]+b[i];
}
● Using vector operations (add, mul, div, ...)
● Two versions
– True SIMD: n Arithmetic Unit
I. Basics

● n highlevel operations at a time
– Pipelines SIMD: Arithmetic Pipelined (depth n)
● n nonidentical lowlevel operations at a time


Real Life Examples
● Cray: pipelined SIMD vector supercomputers
● Modern MicroProcessor Instruction Set
contains SIMD instructions
– Intel: MMX, SSE{1,2,3}
– AMD: 3dnow
– IBM: Altivec
– Sun: VSI
I. Basics

● Very efficient
– 2002: winner of the TOP500 list was the Japanese
Earth Simulator, a vector supercomputer


MIMD Architecture
● Two forms
– Shared memory: any processor can access any
memory region
– Distributed memory: a distinct memory is
associated with each processor
● Usage:
– Multiple sequential programs can be executed
simultaneously
I. Basics

– Different parts of a program are executed
simultaneously on each processor
● Need cooperation!


Warning: Definitions
● Multiprocessors: computers capable of
running multiple instructions streams
simultaneously to cooperatively execute a
single program
● Multiprogramming: sharing of computing
resources by independent jobs
● Multiprocessing:
– running a possibly sequential program on a
I. Basics

multiprocessor (not our concern here)
– running a program consisting of multiple
cooperating processes


Shared vs Distributed

M
CPU CPU CPU
CPU

Switch M CPU Switch CPU M

Memory
I. Basics

Shared Memory Distributed Memory


MIMD
Cooperation/Communication
● Shared memory
– read/write in the shared memory by hardware
– synchronisation with locking primitives
(semaphore, monitor, etc.)
– Variable Latency
● Distributed memory
– explicit message passing in software
I. Basics

● synchronous, asynchronous, grouped, remote method
invocation
– Large message may mask long and variable latency


Interconnection Networks
I. Basics


“Hard” Network Metrics
● Bandwidth (B): maximum number of bytes the
network can transport
● Latency (L): transmission time of a single
message
– 4 components
● software overhead (so): msg packing/unpacking
● routing delay (rd): time to establish the route
● contention time (ct): network is busy
I. Basics

● channel delay (cd): msg size/bandwidth
– so+ct depends on program behaviour
– rd+cd depends on hardware

“Soft” Network Metrics
● Diameter (r): longest path between two nodes
r
Average Distance (da): 1
∑ d.N d
●

N −1 d =1
where d is the distance between two nodes defined as the number of links in
the shortest path between them, N the total number of nodes and Nd , the
number of nodes with a distance d from a given node.
● Connectivity (P): the number of nodes that can
be reached from a given node in one hop.
Concurrency (C): the number of independent
I. Basics

●

connections that a network can make.
– bus: C = 1; linear: C = N1(if processor can send and
receive simultaneously, otherwise, C = ⌊N/2⌋)

Influence of the network
● Bus
– low cost,
– low concurrency degree: 1 message at a time
● Fully connected
– high cost: N(N1)/2 links
– high concurrency degree: N messages at a time
● Star
I. Basics

– Low cost (high availability)
– Exercice: Concurrency Degree?


Parallel Responsibilities

Human writes a Parallel Algorithm
Human implements using a Language
Compiler translates into a Parallel Program
Runtime/Libraries are required in a Parallel Process
Human & Operating System map to the Hardware
The hardware is leading the Parallel Execution

Many actors are involved in the execution We usually consider
I. Basics

of a program (even sequential). explicit parallelism:
Parallelism can be introduced when parallelism is
automatically by compilers, introduced at the
runtime/libraries, OS and in the hardware language level


SIMD Pseudo-Code
● SIMD
<indexed variable> = <indexed expression>, (<index range>);
Example:
C[i,j] == A[i,j] + B[i,j], (0 ≤ i ≤ N-1)

j j j
i i i
C = A + B

j=0 j=1 j = N-1
I. Basics

= + = + ... = +
C[i] A[i] B[i] C[i] A[i] B[i] C[i] A[i] B[i]

SIMD Matrix Multiplication
● General Formula
N −1
cij = ∑ a ik b kj
k =0

for (i = 0; i < N; i++) {
// SIMD Initialisation
C[i,j] = 0, (0 <= j < N);
for (k = 0; k < N; k++) {
I. Basics

// SIMD computation
C[i,j]=C[i,j]+A[i,k]*B[k,j], (0 <= j < N);
}
}


SIMD C Code (GCC)
● gcc vector type & operations
● Vector of 8 bytes long, composed of 4 short elements
typedef short v4hi __attribute__ ((vector_size (8)));
● Use union to access individual elements
union vec { v4hi v, short s[4]; };
● Usual operations on vector variables
v4hi v, w, r; r = v+w*w/v;

● gcc builtins functions
● #include <mmintrin.h> // MMX only
I. Basics

● Compile with mmmx
● Use the provided functions
_mm_setr_pi16 (short w0, short w1, short w2, short w3);
_mm_add_pi16 (__m64 m1, __m64 m2);
...


MIMD Shared Memory
Pseudo Code
● New instructions:
– fork <label> Start a new process executing at
<label>;
– join <integer> Join <integer> processes into one;
– shared <variable list> make the storage class of the
variables shared;
– private <variable list> make the storage class of the
variables private.
I. Basics

● Sufficient? What about synchronization?
– We will focus on that later on...


MIMD Shared Memory Matrix
Multiplication
● private i,j,k;
shared A[N,N], B[N,N], C[N,N], N;
for (j = 0; j < N - 2; j++) fork DOCOL;
// Only the Original process reach this point
j = N-1;
DOCOL: // Executed by N process for each column
for (i = 0; i < N; i++) {
C[i,j] = 0;
for (k = 0; k < N; k++) {
// SIMD computation
C[i,j]=C[i,j]+A[i,k]*B[k,j];
I. Basics

}
}
join N; // Wait for the other process


MIMD Shared Memory C Code
● Process based:
– fork()/wait()
● Thread Based:
– Pthread library
– Solaris Thread library
● Specific Framework: OpenMP
● Exercice: read the following articles:
I. Basics

– “Imperative Concurrent ObjectOriented Languages”,
M.Philippsen, 1995
– “Threads Cannot Be Implemented As a Library”,
HansJ.Boehm, 2005


MIMD Distributed Memory
Pseudo Code
● Basic instructions send()/receive()
● Many possibilities:
– nonblocking send needs message buffering
– nonblocking receive needs failure return
– blocking send and receive both needs termination
detection
● Most used combinations
blocking send and receive
I. Basics

●

● nonblocking send and blocking receive
● Grouped communications
– scatter/gather, broadcast, multicast, ...

MIMD Distributed Memory
C Code
● Traditional Sockets
– Bad Performance
– Grouped communication limited
● Parallel Virtual Machine (PVM)
– Old but good and portable
● Message Passing Interface (MPI)
– The standard used in parallel programming today
I. Basics

– Very efficient implementations
● Vendors of supercomputers provide their own highly
optimized MPI implementation
● Opensource implementation available

Performance

●
Size and Depth
Speedup and Efficiency
Outline

●


Parallel Program
Characteristics
● s = V[0];
for (i = 1; i < N; i++) {
s = s + V[i];
}
V[0] V[1] V[2] V[3] ... V[N-1]
I. Performance

+
+
Data Dependence Graph
● Size = N-1 (nb op)
+
● Depth = N-1 (nb op in the

longest path from +
any input to any output)
s

Parallel Program
Characteristics
V[0] V[1] V[2] V[3] V[N-2] V[N-1]

+ + +
I. Performance

+ +

+
● Size = N-1
● Depth =⌈log2N⌉
s


Parallel Program
Characteristics
● Size and Depth does not take into account
many things
– memory operation (allocation, indexing, ...)
– communication between processes
● read/write for shared memory MIMD
I. Performance

● send/receive for distributed memory MIMD
– Load balancing when the number of processors is
less than the number of parallel task
– etc...
● Inherent Characteristics of a Parallel Machine
independent of any specific architecture

Prefix Problem
● for (i = 1; i < N; i++) {
V[i] = V[i-1] + V[i];
}
V[0] V[1] V[2] V[3] ... V[N-1]

+
● Size = N-1
I. Performance

● Depth =N-1
+

+
How to make this
+ program parallel?

V[0] V[1] V[2] V[3] ... V[N-1]

Prefix Solution 1
Upper/Lower Prefix Algorithm
● Divide and Conquer
– Divide a problem of size N into 2 equivalent
problems of size N/2
Apply the same algorithm
V[0] V[N-1] for sub-prefix.
... ... When N=2, use the direct,
sequential implementation
⌈N/2⌉ Prefix ⌊N/2⌋ Prefix
Size has increased!
N-2+N/2 > N-1 (N>2)

Depth = ⌈log2N⌉
... + + ... +
Size (N=2k) = k.2k-1
=(N/2).log2N
V[0] V[N-1]

ul
Analysis of P
● Proof by induction
● Only for N a power of 2
– Assume from construction that size increase
smoothly with N
● Example: N=1,048,576=220
– Sequential Prefix Algorithm:
● 1,048,575 operations in 1,048,575 time unit steps
– Upper/Lower Prefix Algorithm
● 10,485,760 operations in 20 time unit steps


Other Prefix Parallel
Algorithms
● Pul requires N/2 processors available!
– In our example, 524,288 processors!
● Other algorithms (exercice: study them)
– Odd/Even Prefix Algorithm
● Size = 2Nlog2(N)2 (N=2k)
● Depth = 2log2(N)2
– Ladner and Fischer's Parallel Prefix
● Size = 4N4.96N0.69 + 1
● Depth = log2(N)
● Exercice: implement them in C

Benchmarks
(Inspired by Pr. Ishfaq Ahmad Slides)

● A benchmark is a performance testing program
– supposed to capture processing and data
movement characteristics of a class of applications.
● Used to measure and to predict performance
of computer systems, and to reveal their
architectural weakness and strong points.
● Benchmark suite: set of benchmark programs.
● Benchmark family: set of benchmark suites.
● Classification:
– scientific computing, commercial, applications,
network services, multimedia applications, ...


Benchmark Examples

Type Name Measuring
Numerical computing (linear al-
LINPACK
gebra)
Micro-Benchmark System calls and data movement
LMBENCH
operations in Unix
STREAM Memory bandwidth
NAS Parallel computing (CFD)
PARKBENCH Parallel computing
SPEC A mixed benchmark family
Macro-Benchmark
Splash Parallel computing
STAP Signal processing
TPC Commercial applications


SPEC Benchmark Family
● Standard Performance Evaluation Corporation
– Started with benchmarks that measure CPU
performance
– Has extended to client/server computing,
commercial applications, I/O subsystems, etc.
● Visit http://www.spec.org/ for more
informations


Performance Metrics
● Execution time
– Generally in seconds
– Real time, User Time, System time?
● Real time
● Instruction Count
– Generally in MIPS or BIPS
– Dynamic: number of executed instructions, not the
number of assembly line code!
● Number of Floating Point Operations Executed
– Mflop/s, Gflop/s
– For scientific applications where flop dominates

Memory Performance
● Three parameters:
– Capacity,
– Latency,
– Bandwidth


Parallelism and Interaction
Overhead
● Time to execute a parallel program is
T = Tcomp+Tpar+Tinter
– Tcomp: time of the effective computation
– Tpar: parallel overhead
– Process management(creation, termination, context switching)
– Grouping operations (creation, destruction of group)
– Process Inquiry operation (Id, Group Id, Group size)
– Tinter: interaction overhead
– Synchronization (locks, barrier, events, ...)
– Aggregation (reduction and scan)
– Communication (p2p, collective, read/write shared variables)


Measuring Latency
Ping-Pong
for (i=0; i < Runs; i++)
if (my_node_id == 0) { /* sender */
tmp = Second();
start_time = Second();
send an m-byte message to node 1;
receive an m-byte message from node 1;
end_time = Second();
timer_overhead = start_time - tmp ;
total_time = end_time - start_time - timer_overhead;
communication_time[i] = total_time / 2 ;
} else if (my_node_id ==1) {/* receiver */
receive an m-byte message from node 0;
send an m-byte message to node 0;
}
}


Measuring Latency
Hot-Potato (fire-brigade)
Method
● n nodes are involved (instead of just two)
● Node 0 sends an mbytes message to node 1
● On receipt, node 1 immediately sends the
same message to node 2 and so on.
● Finally node n1 sends the message back to
node 0
● The total time is divided by n to get the point
topoint average communication time.


Collective Communication
Performance
for (i = 0; i < Runs; i++) {
barrier synchronization;
tmp = Second();
start_time = Second();
for (j = 0; j < Iterations; j++)
the_collective_routine_being_measured;
end_time = Second();
timer_overhead = start_time - tmp;
total_time = end_time - start_time - timer_overhead;
local_time = total_time / Iterations;
communication_time[i] = max {all n local_time};
}

How to get all n local time?


Point-to-Point
Communication Overhead
● Linear function of the message length m (in
bytes)
– t(m) = t0 + m/r∞
– t0: startup time
– r∞: asymptotic bandwidth
t
● Ideal Situation!
● Many things are left out
● Topology metrics
t0
● Network Contention

m

● Broadcast: 1 process sends an mbytes message to all
n process
● Multicast: 1 process sends an mbytes message to p <
n process
● Gather: 1 process receives an mbytes from each of
the n process (mn bytes received)
● Scatter: 1 process sends a distinct mbytes message to
each of the n process (mn bytes sent)
● Total exchange: every process sends a distinct m
bytes message to each of the n process (mn2 bytes
sent)
● Circular Shift: process i sends an mbytes message to
process i+1 (process n1 to process 0)

Overhead
● Function of both m and n
– startup latency depends only on n
T(m,n) = t0(n) + m/r∞(n)
● Many problems here! For a broadcast for example,
how is done the communication at the link layer?
● Sequential?

● Process 1 sends 1 m-bytes message to process 2,

then to process 3, and so on... (n steps required)
● Truly Parallel (1 step required)

● Tree based (log (n) steps required)
2
● Again, many things are left out
● Topology metrics

● Network Contention


Speedup
(Problem-oriented)

● For a given algorithm, let Tp be time to perform
the computation using p processors
– T∞: depth of the algorithm
● How much time we gain by using a parallel
algorithm?
– Speedup: Sp=T1/Tp
– Issue: what is T1?
● Time taken by the execution of the best sequential
algorithm on the same input data
● We note T1=Ts


Speedup consequence

● Best Parallel Time Achievable is: Ts/p
● Best Speedup is bounded by Sp≤Ts/(Ts/p)=p
– Called Linear Speedup
● Conditions
– Workload can be divided into p equal parts
– No overhead at all
● Ideal situation, usually impossible to achieve!
● Objective of a parallel algorithm
– Being as close to a linear speedup as possible


Biased Speedup
(Algorithm-oriented)
● We usually don't know what is the best
sequential algorithm except for some simple
problems (sorting, searching, ...)
● In this case, T1
– is the time taken by the execution of the same
parallel algorithm on 1 processor.
● Clearly biased!
– The parallel algorithm and the best sequential
algorithm are often very, very different
● This speedup should be taken with that
limitation in mind!

Biased Speedup Consequence
● Since we compare our parallel algorithm on p
processors with a suboptimal sequential
(parallel algorithm on only 1 processor) we can
have: Sp>p
– Superlinear speedup
● Can also be the consequence of using
– a unique feature of the system architecture that
favours parallel formation
– indeterminate nature of the algorithm
– Extramemory due to the use of p computers (less
swap)

Efficiency
● How long processors are being used on the
computation?
– Efficiency: Ep=Sp/p (expressed in %)
– Also represents how much benefits we gain by
using p processors
● Bounded by 100%
– Superlinear speedup of course can give more than
100% but it is biased


Amdhal's Law: fixed problem
size
(1967)
● For a problem with a workload W, we assume
that we can divide it into two parts:
– W = W + (1)W
 percent of W must be executed sequentially , and the
remaining 1 can be executed by n nodes simultaneously with
100% of efficiency
– Ts(W)=Ts(W+(1)W)=Ts(W)+Ts((1)W)=Ts(W)+(1)Ts(W)

– Tp(W)=Tp(W+(1)W)=Ts(W)+Tp((1)W)=Ts(W)+(1)Ts(W)/p

Ts p 1
S p= =   p ∞
1−T s 1 p−1 
 T s
p

Amdhal's Law: fixed problem
size
Ts
αTs (1-α)Ts
serial section parallelizable sections
...

P processors
...

(1-α)Ts/p
Tp

Amdhal's Law Consequences

S(p) S(α)

p=100
α=5%

α=10%

α=20%
p=10
α=40%


● For a fixed workload, and without any
overhead, the maximal speedup as an upper
bound of 1/
● Taking into account the overhead To
– Performance is limited not only by the sequential
bottleneck but also by the average overhead
Ts p
S p= =
1−T s pT o
 T s T o 1 p−1
p Ts
1
S p  p  ∞
To

Ts

● The sequential bottleneck cannot be solved
just by increasing the number of processors in
a system.
– Very pessimistic view on parallel processing during
two decades .
● Major assumption: the problem size
(workload) is fixed.

Conclusion don't use parallel
programming on a small sized problem!


Gustafson's Law: fixed time
(1988)
● As the machine size increase, the problem size
may increase with execution time unchanged
● Sequential case: W=W+(1)W
– Ts(W)=Ts(W)+(1)Ts(W)

– Tp(W)=Ts(W)+(1)Ts(W)/p

– To have a constant time, the parallel workload
should be increased
● Parallel case (pprocessor): W'=W+p(1)W.
– Ts(W')= Ts(W)+p(1)Ts(W)
● Assumption: sequential part is constant: it does not increase
with the size of the problem


Fixed time speedup

T s W '  T s W '   T s W  p 1−T s W 
S  p= = =
T p W '  T s W   T s W 1−T s W 

S  p=1− p

Fixed time speedup is a linear function of p,
if the workload is scaled up
to maintain a fixed execution time


Gustafson's Law
Consequences
α=5% p=100
S(p) S(α)
p=50

α=10%
α=20% p=25

α=40% p=10


Fixed time speedup with
overhead
T s W '  T s W '  1− p
S  p= = =
T p W '  T s W T o  p T o  p
1
T s W 

● Best fixed speedup is achieved with
a low overhead as expected
● T : function of the number of processors
o
● Quite difficult in practice to make it a

decreasing function


SIMD Programming

●
Matrix Addition & Multiplication
Gauss Elimination
Outline

●


Matrix Vocabulary
● A matrix is dense if most of its elements are
nonzero
● A matrix is sparse if a significant number of its
elements are zero
– Space efficient representation available
– Efficient algorithms available
● Usual matrix size in parallel programming
– Lower than 1000x1000 for a dense matrix
– Bigger than 1000x1000 for a sparse matrix
– Growing with the performance of computer
architectures

Matrix Addition

C = AB , c i , j =a i , j bi , j 0≤i≤n , 0≤ j≤m

// Sequential
for (i = 0; i < n; i++) { Need to map the vectored
for (j = 0; j < m; j++) { addition of m elements a[i]+b[i]
c[i,j] = a[i,j]+b[i,j]; to our underlying architecture
} made of p processing elements
}

// SIMD
Complexity:
for (i = 0; i < n; i++) { – Sequential case: n.m additions
c[i,j] = a[i,j]+b[i,j]; – SIMD case: n.m/p
(0 <= j < m)
Speedup: S(p)=p
}


Matrix Multiplication

C = AB , A[n , l ] ; B [l , m] ;C [n , m]
l −1
ci , j = ∑ a i , k b k , j 0≤i≤n ,0≤ j≤m
k =0

for (i = 0; i < n; i++) {
for (j = 0; j < m; j++) {
c[i,j] = 0;
for (k = 0; k < l; k++) {
c[i,j] = c[i,j] + a[i,k]*b[k,j];
}
}
Complexity:
– n.m initialisations
}
– n.m.l additions and multiplications
Overhead: Memory indexing!
– Use a temporary variable sum for c[i,j]!


SIMD Matrix Multiplication

for (i = 0; i < n; i++) {
c[i,j] = 0; (0 <= j < m) // SIMD Op
for (k = 0; k < l; k++) {
c[i,j] = c[i,j] + a[i,k]*b[k,j];
(0 <= j < m)
}
} Complexity (n = m = l for simplicity):
– n2/p initialisations (p<=n)
} 3
– n /p additions and multiplications (p<=n)
Overhead: Memory indexing!
– Use a temporary vector for c[i]!

Algorithm in O(n) with p=n2
– Assign one processor for
Algorithm in O(lg(n)) with p=n3
each element of C – Parallelize the inner loop!
– Initialisation in O(1)
– Lowest bound
– Computation in O(n)


Gauss Elimination
● Matrix (n x n) represents a system of n linear
equations of n unknowns
a n−1,0 x 0a n−1,1 x 1 a n−1,2 x 2 ...a n−1, n−1 x n−1 =b n−1
a n−2,0 x 0a n−2,1 x 1a n−2,2 x 2 ...a n−2, n−1 x n−1 =b n−2
...
a 1,0 x 0a 1,1 x 1 a 1,2 x 2...a 1, n−1 x n−1 =b1
a 0,0 x 0 a 0,1 x 1 a 0,2 x 2...a 0, n−1 x n−1=b0

 
a n−1,0 a n−1,1 a n−1,2 ... a n−1, n−1 b n−1
a n−2,0 a n−2,1 a n−2,2 ... a n−2, n−1 bn−2
Represented by M = ... ... ... ... ... ...
a 1,0 a 1,1 a 1,2 ... a 1, n−1 b1
a 0,0 a 0,1 a 0,2 ... a 0, n−1 b0

Gauss Elimination Principle
● Transform the linear system into a triangular
system of equations.

At the ith row, each row j below is replaced by:
{row j} + {row i}.(-aj, i/ai, i)


Gauss Elimination
Sequential & SIMD
Algorithms
for (i=0; i<n-1; i++){ // for each row except the last
for (j=i+1; j<n; j++) // step through subsequent rows
m=a[j,i]/a[i,i]; // compute multiplier
for (k=i; k<n; k++) {
a[j,k]=a[j,k]-a[i,k]*m;
} Complexity: O(n3)
}
}

for (i=0; i<n-1; i++){ SIMD Version Complexity: O(n3/p)
for (j=i+1; j<n; j++)
m=a[j,i]/a[i,i];
a[j,k]=a[j,k]-a[i,k]*m; (i<=k<n)
}
}

Other SIMD Applications
● Image Processing
● Climate and Weather Prediction
● Molecular Dynamics
● SemiConductor Design
● Fluid Flow Modelling
● VLSIDesign
● Database Information Retrieval
● ...


MIMD SM Programming

●
Process, Threads & Fibers
●
Pre- & self-scheduling
Common Issues
Outline

●

●
OpenMP


Process, Threads, Fibers
● Process: heavy weight
– Instruction Pointer, Stacks, Registers
– memory, file handles, sockets, device handles, and
windows
● (kernel) Thread: medium weight
– Known and scheduled preemptively by the kernel
● (user thread) Fiber: light weight
– Unkwown by the kernel and scheduled
cooperatively in user space

Multithreaded Process

Code Data Files Code Data Files

Reg Reg Reg
Registers Stack
Stack Stack Stack
thread

Process Process
Single-Threaded Multi-Threaded

Kernel Thread
● Provided and scheduled by the kernel
– Solaris, Linux, Mac OS X, AIX, Windows XP
– Incompatibilities: POSIX PThread Standard
● Scheduling Question: Contention Scope
– The time slice q can be given to:
● the whole process (PTHREAD_PROCESS_SCOPE)
● Each individual thread (PTHREAD_SYSTEM_SCOPE)
● Fairness?
– With a PTHREAD_SYSTEM_SCOPE, the more
threads a process has, the more CPU it has!


Fibers (User Threads)
● Implemented entirely in user space (also called
green theads)
● Advantages:
– Performance: no need to cross the kernel (context
switch) to schedule another thread
– Scheduler can be customized for the application
● Disadvantages:
– Cannot directly use multiprocessor
– Any system call blocks the entire process
● Solutions: Asynchronous I/O, Scheduler Activation
● Systems: NetBSD, Windows XP (FiberThread)

Mapping
● Threads are expressed by the application in
user space.
● 11: any thread expressed in user space is
mapped to a kernel thread
– Linux, Windows XP, Solaris > 8
● n1: n threads expressed in user space are
mapped to one kernel thread
– Any userspace only thread library
● nm: n threads expressed in user space are
mapped to m kernel threads (n>>m)
– AIX, Solaris < 9, IRIX, HPUX, TRU64 UNIX

Mapping
User Space User Space User Space

Kernel Space Kernel Space Kernel Space
1-1 n-1 n-m


MIMD SM Matrix Addition
Self-Scheduling
● private i,j,k;
shared A[N,N], B[N,N], C[N,N], N;
for (i = 0; i < N - 2; i++) fork DOLINE;
// Only the Original process reach this point
i = N-1; // Really Required?
DOLINE: // Executed by N process for each line
for (j = 0; j < N; j++) {
// SIMD computation
C[i,j]=A[i,j]+B[i,j];Complexity: O(n2/p)
} If SIMD is used: O(n2/(p.m))
} m:number of SIMD PEs
join N; // Wait for the other process
Issue: if n > p, self-scheduling.
The OS decides who will do what. (Dynamic)
In this case, overhead


MIMD SM Matrix Addition
Pre-Scheduling
● Basic idea: divide the whole matrix by p=s*s
submatrix
submatrices
s

C = A + B
s

Each of the p threads compute:
= +
Ideal Situation! No communication Overhead!
Pre-Scheduling: decide statically who will do what.

Parallel Primality Testing
(from Herlihy and Shavit Slides)

● Challenge
– Print primes from 1 to 1010
● Given
– Tenprocessor multiprocessor
– One thread per processor
● Goal
– Get tenfold speedup (or close)


Load Balancing
● Split the work evenly
● Each thread tests range of 109

1 109 2·109 … 1010

P0 P1 … P9
Code for thread i:
void thread(int i) {
for (j = i*109+1, j<(i+1)*109; j++) {
if (isPrime(j)) print(j);
}
}


Issues
● Larger Num ranges have fewer primes
● Larger numbers harder to test
Thread workloads
ed
●

– Uneven
je ct
– Hard to predict re
● Need dynamic load balancing


Shared Counter

19
each thread takes a number
18

17


Procedure for Thread i

// Global (shared) variable
static long value = 1;

void thread(int i) {
long j = 0;
while (j < 1010) {
j = inc();
if (isPrime(j)) print(j);
}
}


Counter Implementation

static long value = 1; r,
so
oces sor
nipr ces
long inc() u
for { ltipro
OK r mu
return value++;
} n o t fo


Counter Implementation


long inc() {
return value++; = value;
temp
} value = value + 1;
return temp;


Uh-Oh

Value… 2 3 2

read 1 read 2
write 2 write 3

read 1 write 2

time


Challenge


long inc() {
temp = value;
value = temp + 1;
Make these steps
return temp;
atomic (indivisib
}


An Aside: C/Posix Threads API

static pthread_mutex_t mutex;
// somewhere before
pthread_mutex_init(&mutex, NULL);

long inc() {
pthread_mutex_lock(&mutex);
temp = value;
Critical section
value = temp + 1;
pthread_mutex_unlock(&mutex);
return temp;
}


Correctness
● Mutual Exclusion
– Never two or more in critical section
– This is a safety property
● No Lockout (lockoutfreedom)
– If someone wants in, someone gets in
– This is a liveness property


Multithreading Issues
● Coordination
– Locks, Condition Variables, Volatile
● Cancellation (interruption)
– Asynchronous (immediate)
● Equivalent to 'kill KILL' but much more dangerous!!
– What happens to lock, open files, buffers, pointers, ...?
● Avoid it as much as you can!!
– Deferred
● Target thread should check periodically if it should be
cancelled
● No guarantee that it will actually stop running!


General Multithreading
Guidelines
● Use good software engineering
– Always check error status
● Always use a while(condition) loop when
waiting on a condition variable
● Use a thread pool for efficiency
– Reduce the number of created thread
– Use thread specific data so one thread can holds its
own (private) data
● The Linux kernel schedules tasks only,
– Process == a task that shares nothing
– Thread == a task that shares everything

General Multithreading
Guidelines
● Give the biggest priorities to I/O bounded
threads
– Usually blocked, should be responsive
● CPUBounded threads should usually have the
lowest priority
● Use Asynchronous I/O when possible, it is
usually more efficient than the combination of
threads+ synchronous I/O
● Use eventdriven programming to keep
indeterminism as low as possible
● Use threads when you really need it!

OpenMP
● Design as superset of common languages for
MIMDSM programming (C, C++, Fortran)
● UNIX, Windows version
● fork()/join() execution model
● OpenMP programs always start with one thread
● relaxedconsistency memory model
● each thread is allowed to have its own temporary view of
the memory
● Compiler directives in C/C++ are called
pragma (pragmatic information)
● Preprocessor directives:
#pragma omp <directive> [arguments]


OpenMP


OpenMP
Thread Creation
● #pragma omp parallel [arguments]
<C instructions> // Parallel region
● A team of thread is created to execute the
parallel region
– the original thread becomes the master thread
(thread ID = 0 for the duration of the region)
– numbers of threads determined by
● Arguments (if, num_threads)
● Implementation (nested parallelism support, dynamic
adjustment)
● Environment variables (OMP_NUM_THREADS)


OpenMP
Work Sharing
● Distributes the execution of a parallel region
among team members
– Does not launch threads
– No barrier on entry
– Implicit barrier on exit (unless nowait specified)
● WorkSharing Constructs
– Sections
– Loop
– Single
– Workshare (Fortran)

OpenMP
Work Sharing - Section

double e, pi, factorial, product;
int i;
e = 1; // Compute e using Taylor expansion
factorial = 1;
for (i = 1; i<num_steps; i++) {
factorial *= i;
e += 1.0/factorial;
} // e done
pi = 0; // Compute pi/4 using Taylor expansion
for (i = 0; i < num_steps*10; i++) {
pi += 1.0/(i*4.0 + 1.0);
pi -= 1.0/(i*4.0 + 3.0);
} Independent Loops
pi = pi * 4.0; // pi done Ideal Case!
return e * pi;


OpenMP
double e, pi, factorial, product; int i; Private Copies
#pragma omp parallel sections shared(e, pi) { Except for
#pragma omp section { shared variables
e = 1; factorial = 1;
for (i = 1; i<num_steps; i++) {
factorial *= i; e += 1.0/factorial;
}
} /* e section */
Independent
#pragma omp section {
sections are
pi = 0; declared
for (i = 0; i < num_steps*10; i++) {
pi += 1.0/(i*4.0 + 1.0);
pi -= 1.0/(i*4.0 + 3.0);
}
pi = pi * 4.0;
} /* pi section */ Implicit Barrier
} /* omp sections */ on exit
return e * pi;

Talyor Example Analysis
– OpenMP Implementation: Intel Compiler
– System: Gentoo/Linux2.6.18
– Hardware: Intel Core Duo T2400 (1.83 Ghz) Cache
L2 2Mb (SmartCache) Reason: Core Duo
Gcc: Architecture?
Standard: 13250 ms,
Intel CC:
Standard: 12530 ms
Optimized (-Os -msse3
Optimized (-O2 -xW
-march=pentium-m
-march=pentium4): 7040 ms
-mfpmath=sse): 7690 ms

OpenMP Standard: 15340 ms Speedup: 0.6
OpenMP Optimized: 10460 ms Efficiency: 3%


OpenMP
● Define a set of structured blocks that are to be
divided among, and executed by, the threads
in a team
● Each structured block is executed once by one
of the threads in the team
● The method of scheduling the structured
blocks among threads in the team is
implementation defined
● There is an implicit barrier at the end of a
sections construct, unless a nowait clause is
specified

OpenMP
Work Sharing - Loop
// LU Matrix Decomposition
for (k = 0; k<SIZE-1; k++) {
for (n = k; n<SIZE; n++) {
col[n] = A[n][k];
}
for (n = k+1; n<SIZE; n++) {
A[k][n] /= col[k];
}
for (n = k+1; n<SIZE; n++) {
row[n] = A[k][n];
}
for (i = k+1; i<SIZE; i++) {
for (j = k+1; j<SIZE; j++) {
A[i][j] = A[i][j] - row[i] * col[j];
}
}
}


OpenMP
Work Sharing - Loop
// LU Matrix Decomposition
for (k = 0; k<SIZE-1; k++) {
... // same as before
#pragma omp parallel for shared(A, row, col)
for (i = k+1; i<SIZE; i++) {
for (j = k+1; j<SIZE; j++) {
A[i][j] = A[i][j] - row[i] * col[j];
}
}
}

Loop should be in
Private Copies canonical form
Except for Implicit Barrier (see spec)
shared variables at the end of a loop
(Memory Model)


OpenMP
Work Sharing - Loop
● Specifies that the iterations of the associated
loop will be executed in parallel.
● Iterations of the loop are distributed across
threads that already exist in the team
● The for directive places restrictions on the
structure of the corresponding forloop
– Canonical form (see specifications):
● for (i = 0; i < N; i++) {...} is ok
● For (i = f(); g(i);) {... i++} is not ok
● There is an implicit barrier at the end of a loop
construct unless a nowait clause is specified.

OpenMP
Work Sharing - Reduction
●
reduction(op: list)
e.g: reduction(*: result)
● A private copy is made for each variable
declared in 'list' for each thread of the
parallel region
● A final value is produced using the operator
'op' to combine all private copies
#pragma omp parallel for reduction(*: res)
for (i = 0; i < SIZE; i++) {
res = res * a[i];
}
}


OpenMP
Synchronisation -- Master
● Master directive:
#pragma omp master
{...}
● Define a section that is only to be executed by
the master thread
● No implicit barrier: other threads in the team
do not wait


OpenMP
Synchronisation -- Critical
● Critical directive:
#pragma omp critical [name]
{...}
● A thread waits at the beginning of a critical
region until no other thread is executing a
critical region with the same name.
● Enforces exclusive access with respect to all
critical constructs with the same name in all
threads, not just in the current team.
● Constructs without a name are considered to
have the same unspecified name.

OpenMP
Synchronisation -- Barrier
#pragma omp barrier
{...}
– All threads of the team must execute the barrier
before any are allowed to continue execution
– Restrictions
● Each barrier region must be encountered by all threads
in a team or by none at all.
● Barrier in not part of C!
– If (i < n)
#pragma omp barrier
// Syntax error!


OpenMP
Synchronisation -- Atomic
#pragma omp atomic
expr-stmt
●
Where expr-stmt is:
x binop= expr, x++, ++x, x--, --x (x scalar, binop any non
overloaded binary operator: +,/,-,*,<<,>>,&,^,|)
● Only load and store of the object designated
by x are atomic; evaluation of expr is not
atomic.
● Does not enforce exclusive access of same
storage location x.
● Some restrictions (see specifications)

OpenMP
Synchronisation -- Flush
#pragma omp flush [(list)]
● Makes a thread’s temporary view of memory
consistent with memory
● Enforces an order on the memory operations
of the variables explicitly specified or implied.
● Implied flush:
– barrier, entry and exit of parallel region, critical
– Exit from work sharing region (unless nowait)
– ...
● Warning with pointers!

MIMD DM Programming

●
Message Passing Solutions
●
Process Creation
Send/Receive
Outline

●

●
MPI


Message Passing Solutions
● How to provide Message Passing to
programmers?
– Designing a special parallel programming language
● Sun Fortress, E language, ...
– Extending the syntax of an existing sequential
language
● ProActive, JavaParty
– Using a library
● PVM, MPI, ...


Requirements
● Process creation
– Static: the number of processes is fixed and defined
at startup (e.g. from the command line)
– Dynamic: process can be created and destroyed at
will during the execution of the program.
● Message Passing
– Send and receive efficient primitives
– Blocking or Unblocking
– Grouped Communication


Process Creation
Multiple Program, Multiple Data
Model
(MPMD)
Source Source
File File
Compile to
suit processor

Executable Executable
File File

Processor 0 Processor p-1


Process Creation
Single Program, Multiple Data Model
(SPMD)
Source
File

Compile to
suit processor
Executable Executable
File File

Processor 0 Processor p-1


SPMD Code Sample

int pid = getPID();
if (pid == MASTER_PID) { ● Compilation for
execute_master_code();
heterogeneous
}else{
execute_slave_code(); processors still required
}

● Master/Slave Architecture
● GetPID() is provided by the
parallel system (library, language, ...)
● MASTER_PID should be well defined


Dynamic Process Creation
Process 1
Primitive required:
: spawn(location, executable)
:
:
spawn() Process 2
: :
: :
: :
: :
:
:
:


Send/Receive: Basic
● Basic primitives:
– send(dst, msg);
– receive(src, msg);
● Questions:
– What is the type of: src, dst?
● IP:port > Low performance but portable
● Process ID > Implementation dependent (efficient
nonIP implementation possible)
– What is the type of: msg?
● Packing/Unpacking of complex messages
● Encoding/Decoding for heterogeneous architectures

Persistence and Synchronicity in Communication (3)
(from Distributed Systems -- Principles and Paradigms, A. Tanenbaum, M. Steen)

2-22.1

a) Persistent asynchronous communication
b) Persistent synchronous communication


2-22.2

a) Transient asynchronous communication
b) Receipt-based transient synchronous communication


a) Delivery-based transient synchronous communication at message
delivery
b) Response-based transient synchronous communication

Send/Receive: Properties

Message System Precedence
Synchronization Requirements Constraints
Send: non-blocking Message Buffering None, unless
Receive: non- Failure return from message is received
blocking receive successfully

Send: non-blocking Message Buffering Actions preceding
send occur before
Receive: blocking Termination Detection those following send
Send: blocking Termination Detection Actions preceding
Receive: non- Failure return from send occur before
blocking receive those following send
Send: blocking Termination Detection Actions preceding
send occur before
Receive: blocking Termination Detection those following send

Receive Filtering
● So far, we have the primitive:
receive(src, msg);
● And if one to receive a message:
– From any source?
– From a group of process only?
● Wildcards are used in that cases
– A wildcard is a special value such as:
● IP: broadast (e.g: 192.168.255.255), multicast address
● The API provides well defined wildcard (e.g:
MPI_ANY_SOURCE)


The Message-Passing Interface (MPI)

Primitive Meaning

MPI_bsend Append outgoing message to a local send buffer

MPI_send Send a message and wait until copied to local or remote buffer

MPI_ssend Send a message and wait until receipt starts

MPI_sendrecv Send a message and wait for reply

MPI_isend Pass reference to outgoing message, and continue

MPI_issend Pass reference to outgoing message, and wait until receipt starts

MPI_recv Receive a message; block if there are none

MPI_irecv Check if there is an incoming message, but do not block

Some of the most intuitive message-passing primitives of MPI.

MPI Tutorial
(by William Gropp)

See William Gropp Slides


Parallel Strategies

●
Embarrassingly Parallel Computation
Partitionning
Outline

●


Embarrassingly Parallel
Computations
● Problems that can be divided into different
independent subtasks
– No interaction between processes
– No data is really shared (but copies may exist)
– Result of each process have to be combined in
some ways (reduction).
● Ideal situation!
● Master/Slave architecture
– Master defines tasks and send them to slaves
(statically or dynamically)


Embarrassingly Parallel
Computations Typical Example:
Image Processing
● Shifting, Scaling, Clipping, Rotating, ...
● 2 Partitioning Possibilities
640 640

480
480

Process Process


Image Shift Pseudo-Code
● Master h=Height/slaves;
for(i=0,row=0;i<slaves;i++,row+=h)
send(row, Pi);
for(i=0; i<Height*Width;i++){
recv(oldrow, oldcol,
newrow, newcol, PANY);
map[newrow,newcol]=map[oldrow,oldcol];
}

recv(row, Pmaster)
● Slave
for(oldrow=row;oldrow<(row+h);oldrow++)
for(oldcol=0;oldcol<Width;oldcol++) {
newrow=oldrow+delta_x;
newcol=oldcol+delta_y;
send(oldrow,oldcol,newrow,newcol,Pmaster);
}

Complexity
● Hypothesis:
– 2 computational steps/pixel
– n*n pixels
– p processes
● Sequential: Ts=2n2=O(n2)
● Parallel: T//=O(p+n2)+O(n2/p)=O(n2) (p fixed)
– Communication: Tcomm=Tstartup+mTdata
=p(Tstartup+1*Tdata)+n2(Tstartup+4Tdata)=O(p+n2)
– Computation: Tcomp=2(n2/p)=O(n2/p)


Speedup
● Speedup Ts 2n
2
=
T p 2 n 2 / p pT startup T data n 2 T startup 4T data 
● Computation/Communication:
2n 2 / p n2 / p
2
=O  2

p T startupT data n T startup 4T data  pn
– Constant when p fixed and n grows!
– Not good: the largest, the better
● Computation should hide the communication
● Broadcasting is better
● This problem is for Shared Memory!


Computing
the
Mandelbrot
Set
●Problem: we consider the limit of the following
sequence in the complex plane: z 0=0
z k 1= z k c
● The constant c=a.i+b is given by the point (a,b)
in the complex plane
● We compute zk for 0<k<n
– if for a given k, (|zk| > 2) we know the sequence
diverge to infinity – we plot c in color(k)
– if for (k == n), |zk|<2 we assume the sequence
converge we plot c in a black color.


Mandelbrot Set: sequential
code
int calculate(double cr, double ci) {
double bx = cr, by = ci;
double xsq, ysq;
int cnt = 0;
Computes the number of iterations
while (true) { for the given complex point (cr, ci)
xsq = bx * bx;
ysq = by * by;
if (xsq + ysq >= 4.0) break;
by = (2 * bx * by) + ci;
bx = xsq - ysq + cr;
cnt++;
if (cnt >= max_iterations) break;
}

return cnt;
}


Mandelbrot Set: sequential
code

y = startDoubleY;
for (int j = 0; j < height; j++) {
double x = startDoubleX;
int offset = j * width;
for (int i = 0; i < width; i++) {
int iter = calculate(x, y);
pixels[i + offset] = colors[iter];
x += dx;
}
y -= dy;
}


Mandelbrot Set:
parallelization
● Static task assignment:
– Cut the image into p areas, and assign them to each
p processor
● Problem: some areas are easier to compute (less
iterations)
– Assign n2/p pixels per processor in a round robin or
random manner
● On average, the number of iterations per processor
should be approximately the same
● Left as an exercice


Mandelbrot set: load
balancing
● Number of iterations per pixel varies
– Performance of processors may also vary
● Dynamic Load Balancing
● We want each CPU to be 100% busy ideally
● Approach: workpool, a collection of tasks
– Sometimes, processes can add new tasks into the
workpool
– Problems:
● How to define a task to minimize the communication
overhead?
● When a task should be given to processes to increase the
computation/communication ratio?

Mandelbrot set: dynamic load
balancing
● Task == pixel
– Lot of communications, Good load balancing
● Task == row
– Fewer communications, less load balanced
● Send a task on request
– Simple, low communication/computation ratio
● Send a task in advance
– Complex to implement, good
communication/computation ratio
● Problem: Termination Detection?


Mandelbrot Set: parallel code
Master Code
count = row = 0;
for(k = 0, k < num_procs; k++) { Pslave is the process
send(row, Pk,data_tag); which sends the last
count++, row++; received message.
}
do {
recv(r, pixels, PANY, result_tag);
count--;
if (row < HEIGHT) {
send(row, Pslave, data_tag);
row++, count++;
}else send(row, Pslave, terminator_tag);
display(r, pixels);
}while(count > 0);


Slave Code

recv(row, PMASTER, ANYTAG, source_tag);
while(source_tag == data_tag) {
double y = startDoubleYFrom(row);
double x = startDoubleX;
for (int i = 0; i < WIDTH; i++) {
int iter = calculate(x, y);
pixels[i] = colors[iter];
x += dx;
}
send(row, pixels, PMASTER, result_tag);
recv(row, PMASTER, ANYTAG, source_tag);
};


Analysis
T s ≤Max iter . n 2
T p =T comm T comp
T comm=n  t startup2 t data  p−1t startup 1t data n  t startupnt data 
n2
T comp ≤ Maxiter
p Only valid when
2
Ts Max iter . n all pixels are black!
=
T p Max iter . n 2 (Inequality of
n2 t data 2 n p−1t startup t data 
p Ts and Tp)
Max iter . n 2
T comp p
= 2
T comm n t data 2 n p−1t startup t data 

● Speedup approaches p when Maxiter is high
● Computation/Communication is in O(Maxiter)


Partitioning
● Basis of parallel programming
● Steps:
– cut the initial problem into smaller part
– solve the smaller part in parallel
– combine the results into one
● Two ways
– Data partitioning
– Functional partitioning
● Special case: divide and conquer
– subproblems are of the same form as the larger
problem

Partitioning Example:
numerical integration
● We want to compute: b

∫ f  x dx
a


Quizz
Outline


Readings
● Describe and compare the following
mechanisms for initiating concurrency:
– Fork/Join
– Asynchronous Method Invocation with futures
– Cobegin
– Forall
– Aggregate
– Life Routine (aka Active Objects)


Parallel Processing

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Parallel Processing (20)

Recently uploaded (20)

Parallel Processing