Collective Communications in MPI

Lecture 4Lecture 4
Collective CommunicationsCollective Communications
Dr. Muhammad Hanif Durad
Department of Computer and Information Sciences
Pakistan Institute Engineering and Applied Sciences
hanif@pieas.edu.pk
Some slides have bee adapted with thanks from some other
lectures available on Internet

Dr. Hanif Durad 2
Lecture Outline
 Collective Communication
 First Program using Collective communication
 The Master- Slave Paradigm
 Multiplying a matrix with a vector
IntroMPI.ppt

Another Approach to Parallelism
 Collective routines provide a higher-level way
to organize a parallel program
 Each process executes the same communication
operations
 MPI provides a rich set of collective
operations…
Dr. Hanif Durad 3
IntroMPI.ppt

Collective Communication
 Involve all processes in the scope of communicator
 Three categories
 synchronization (barrier())
 data movement (broadcast, scatter, gather, alltoall)
 collective computation (reduce(), scan())
 Limitations/differences from point-to-point
 blocking (no more true with MPI 2)
 does not take tag arguments
 works with MPI defined datatypes - not with derived types
Dr. Hanif Durad 4
Comm.ppt

Collective Communication
 Involves set of processes, defined by an intra-communicator. Message
tags not present. Principal collective operations:
 MPI_Bcast() - Broadcast from root to all other processes
 MPI_Gather() - Gather values for group of processes
 MPI_Scatter() - Scatters buffer in parts to group of processes
 MPI_Alltoall() - Sends data from all processes to all processes
 MPI_Reduce() - Combine values on all processes to single value
 MPI_Reduce_scatter() - Combine values and scatter results
 MPI_Scan() - Compute prefix reductions of data on processes
Dr. Hanif Durad 5
slides2.ppt

One to Many
Basic primitives
• broadcast(data, source, group_id, …)
- group members
broadcast(data,…);
source
data
Comm.ppt

One to Many
Basic primitives
• scatter(data[], recvBuf, source, group_id, …)
•gather(sendBuf, recvBuf[], dest, group_id, …)
- group members
source
data
scatter(data,…);
One-to-all personalized communication
Comm.ppt
gather(…)
dest
concatenation
dual of scatter

Many to One
Basic primitives
• reduce (sendBuf, recvBuf, dest, operation, group_id, …)
reduce(… ‘+’…);
- group members
3
dest
10 4
1
0
4
-1
2
1
+
Comm.ppt

Many to One – scan()
Also called parallel prefix:
• scan(sendBuf, recvBuf, operation, group_id, …)
• performs reduce() on all predecessors
scan(sendBuf, recvBuf,‘*’, group_id, …);
- group members
sendBuf4 104-1321
*
recvBuf4 8 -8 -32 -32
* * * *
Prefix-Sum
Comm.ppt

MPI Names of various
Operations
Dr. Hanif Durad 10
Slide from PC course Chapter 4

Calculating the value of π using
mid point integration formula
Dr. Hanif Durad 11
First Program using Collective
communication

Modeling the problem?
Dr. Hanif Durad 12
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4
1 1
2
0
0
1
arctan
1 4
dx x
x
π
=  =
+∫
>> x=0:.1:1
>> y=1+x.^2
>>plot(x,4./y)

Decomposition of a Problem
Dr. Hanif Durad 13

Different Ways to Partition Data
Dr. Hanif Durad 14

Example: PI in C (1/2)
#include "mpi.h"
#include <math.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
int done = 0, n, myid, numprocs, i, rc;
double PI25DT = 3.141592653589793238462643;
double mypi, pi, h, sum, x, a;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
while (!done) {
if (myid == 0) {
printf("Enter the number of intervals: (0 quits) ");
scanf("%d",&n);
}
15Dr. Hanif Durad
pi2.c

Example: PI in C (2/2)
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
if (n == 0) break;
h = 1.0 / (double) n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h * ((double)i - 0.5);
sum += 4.0 / (1.0 + x*x);
}
mypi = h * sum;
MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,
MPI_COMM_WORLD);
if (myid == 0)
printf("pi is approximately %.16f, Error is .16fn",
pi, fabs(pi - PI25DT));
}
MPI_Finalize();
return 0;
} 16Dr. Hanif Durad

Example: PI in Fortran (1/2)
program main
include "mpif.h"
integer done, n, myid, numprocs, i, rc
double pi25dt, mypi, pi, h, sum, x, z
data done /.false./
data PI25DT/3.141592653589793238462643/
call MPI_Init(ierr)
call MPI_Comm_size(MPI_COMM_WORLD,numprocs, ierr )
call MPI_Comm_rank(MPI_COMM_WORLD,myid, ierr)
do while (.not. done)
if (myid .eq. 0) then
print *,"Enter the number of intervals: (0 quits)"
read *, n
endif
call MPI_Bcast(n, 1, MPI_INTEGER, 0, MPI_COMM_WORLD, ierr )
if (n .eq. 0) goto 10 17Dr. Hanif Durad
pi.f90

Example: PI in Fortran (2/2)
h =1.0/n
sum=0.0
do i=myid+1,n,numprocs
x=h*(i - 0.5)
sum+=4.0/(1.0+x*x)
enddo
mypi = h * sum
call MPI_Reduce(mypi, pi, 1, MPI_DOUBLE_PRECISION, MPI_SUM, 0,
MPI_COMM_WORLD, ierr )
if (myid .eq. 0) then
print *, "pi is approximately ", pi, ", Error is ", abs(pi - PI25DT)
endif
10 continue
call MPI_Finalize( ierr )
end
18Dr. Hanif Durad

Example: PI in C++ (1/2)
#include "mpi.h"
#include <math.h>
#include <iostream>
int main(int argc, char *argv[])
{
int done = 0, n, myid, numprocs, i, rc;
double PI25DT = 3.141592653589793238462643;
double mypi, pi, h, sum, x, a;
MPI::Init(argc, argv);
numprocs = MPI::COMM_WORLD.Get_size();
myid = MPI::COMM_WORLD.Get_rank();
while (!done) {
if (myid == 0) {
std::cout << "Enter the number of intervals: (0 quits) ";
std::cin >> n;;
}
19Dr. Hanif Durad
pi.cpp

Example: PI in C++ (2/2)
MPI::COMM_WORLD.Bcast(&n, 1, MPI::INT, 0 );
if (n == 0) break;
h = 1.0 / (double) n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h * ((double)i - 0.5);
sum += 4.0 / (1.0 + x*x);
}
mypi = h * sum;
MPI::COMM_WORLD.Reduce(&mypi, &pi, 1, MPI::DOUBLE,
MPI::SUM, 0);
if (myid == 0)
std::cout << "pi is approximately " << pi <<
", Error is " << fabs(pi - PI25DT) << "n";
}
MPI::Finalize();
return 0;
} 20Dr. Hanif Durad

Notes on C and Fortran
 C and Fortran bindings correspond closely
 In C:
 mpi.h must be #included
 MPI functions return error codes or MPI_SUCCESS
 In Fortran:
 mpif.h must be included, or use MPI module
 All MPI calls are to subroutines, with a place for the return code in the last
argument.
 C++ bindings, and Fortran-90 issues, are part of MPI-2.
Dr. Hanif Durad 21

Multiplying a matrix with a vector
⇒matvec.cpp & matvec1.cpp on
virtue
Dr. Hanif Durad 22
2nd
Program using Collective
communication

The Master- Slave Paradigm
Dr. Hanif Durad 23
Morgan Kaufmann Publishers - Interconnection Networks. An
Engineering Approach.pdf,

Multiplying a matrix with a matrix
Dr. Hanif Durad 26
3rd
Program using Collective
communication

The Collective Programming
Model
 One style of higher level programming is to
use only collective routines
 Provides a “data parallel” style of
programming
 Easy to follow program flow
Dr. Hanif Durad 27

Not Covered
 Topologies: map a communicator onto, say, a 3D Cartesian
processor grid
 Implementation can provide ideal logical to physical mapping
 Rich set of I/O functions: individual, collective, blocking
and non-blocking
 Collective I/O can lead to many small requests being merged for
more efficient I/O
 One-sided communication: puts and gets with various
synchronization schemes
 Task creation and destruction: change number of tasks
during a run
 Few implementations available 28

Collective Communications in MPI

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Collective Communications in MPI (20)

More from Hanif Durad (15)

Recently uploaded (20)

Collective Communications in MPI