SlideShare a Scribd company logo
CS 425 / ECE 428
Distributed Systems
Fall 2023
Indranil Gupta (Indy)
W/ Aishwarya Ganesan
Lecture 16: Multicast
All slides © IG
Multicast Problem
2
• Multicast  message sent to a group of
processes
• Broadcast  message sent to all
processes (anywhere)
• Unicast  message sent from one
sender process to one receiver process
Other Communication Forms
3
• A widely-used abstraction by almost all cloud systems
• Storage systems like Cassandra or a database
– Replica servers for a key: Writes/reads to the key are multicast within the replica group
– All servers: membership information (e.g., heartbeats) is multicast across all servers in
cluster
• Online scoreboards (ESPN, French Open, FIFA World Cup)
– Multicast to group of clients interested in the scores
• Stock Exchanges
– Group is the set of broker computers
– Groups of computers for High frequency Trading
• Air traffic control system
– All controllers need to receive the same updates in the same order
Who Uses Multicast?
4
• Determines the meaning of “same order” of
multicast delivery at different processes in the
group
• Three popular flavors implemented
by several multicast protocols
1. FIFO ordering
2. Causal ordering
3. Total ordering
Multicast Ordering
5
• Multicasts from each sender are received in
the order they are sent, at all receivers
• Don’t worry about multicasts from
different senders
• More formally
– If a correct process issues (sends)
multicast(g,m) to group g and then
multicast(g,m’), then every correct process
that delivers m’ would already have delivered
m.
1. FIFO ordering
6
M1:1 and M1:2 should be received in that order at each receiver
Order of delivery of M3:1 and M1:2 could be different at different receivers
FIFO Ordering: Example
P2
Time
P1
P3
M1:1 M1:2
P4
M3:1
• Multicasts whose send events are
causally related, must be received in the
same causality-obeying order at all
receivers
• Formally
– If multicast(g,m)  multicast(g,m’)
then any correct process that delivers
m’would already have delivered m.
– ( is Lamport’s happens-before)
2. Causal Ordering
8
M3:1  M3:2, and so should be received in that order at each receiver
M1:1  M3:1, and so should be received in that order at each receiver
M3:1 and M2:1 are concurrent and thus ok to be received in different orders at
different receivers
Causal Ordering: Example
P2
Time
P1
P3
M1:1
P4
M3:1 M3:2
M2:1
• Causal Ordering => FIFO Ordering
• Why?
– If two multicasts M and M’ are sent by the same
process P, and M was sent before M’, then M 
M’
– Then a multicast protocol that implements
causal ordering will obey FIFO ordering since
M  M’
• Reverse is not true! FIFO ordering does not
imply causal ordering.
Causal vs. FIFO
10
• Group = set of your friends on a social
network
• A friend sees your message m, and she
posts a response (comment) m’ to it
– If friends receive m’ before m, it wouldn’t
make sense
– But if two friends post messages m” and n”
concurrently, then they can be seen in any
order at receivers
• A variety of systems implement causal
ordering: Social networks, bulletin boards,
comments on websites, etc.
Why Causal at All?
11
• Also known as “Atomic Broadcast”
• Unlike FIFO and causal, this does not pay
attention to order of multicast sending
• Ensures all receivers receive all multicasts in
the same order
• Formally
– If a correct process P delivers message
m before m’ (independent of the
senders), then any other correct
process P’ that delivers m’ would
already have delivered m.
3. Total Ordering
12
The order of receipt of multicasts is the same at all processes.
M1:1, then M2:1, then M3:1, then M3:2
May need to delay delivery of some messages
Total Ordering: Example
P2
Time
P1
P3
M1:1
P4
M3:1 M3:2
M2:1
13
• Since FIFO/Causal are
orthogonal to Total, can have
hybrid ordering protocols too
– FIFO-total hybrid protocol
satisfies both FIFO and total
orders
– Causal-total hybrid protocol
satisfies both Causal and total
orders
Hybrid Variants
14
• That was what ordering is
• But how do we implement
each of these orderings?
Implementation?
15
• Each receiver maintains a per-sender
sequence number (integers)
– Processes P1 through PN
– Pi maintains a vector of sequence
numbers Pi[1…N] (initially all
zeroes)
– Pi[j] is the latest sequence number
Pi has received from Pj
FIFO Multicast: Data Structures
16
• Send multicast at process Pj:
– Set Pj[j] = Pj[j] + 1
– Include new Pj[j] in multicast message as
its sequence number
• Receive multicast: If Pi receives a multicast
from Pj with sequence number S in
message
– if (S == Pi[j] + 1) then
• deliver message to application
• Set Pi[j] = Pi[j] + 1
– else buffer this multicast until above
condition is true
FIFO Multicast: Updating Rules
17
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
FIFO Ordering: Example
18
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
Deliver!
P1, seq: 1
[1,0,0,0]
Deliver!
?
[1,0,0,0]
FIFO Ordering: Example
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
Deliver!
P1, seq: 1
[1,0,0,0]
Deliver!
[0,0,0,0]
Buffer!
P1, seq: 2
[1,0,0,0] [2,0,0,0]
FIFO Ordering: Example
[1,0,0,0]
Deliver this!
Deliver buffered <P1, seq:2>
Update [2,0,0,0]
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
Deliver!
P1, seq: 1
[1,0,0,0]
Deliver!
[0,0,0,0]
Buffer!
P1, seq: 2
[1,0,0,0] [2,0,0,0]
[2,0,0,0]
Deliver!
[1,0,0,0]
Deliver this!
Deliver buffered <P1, seq:2>
Update [2,0,0,0]
FIFO Ordering: Example
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
Deliver!
P1, seq: 1
[1,0,0,0]
Deliver!
[0,0,0,0]
Buffer!
P1, seq: 2
[1,0,0,0] [2,0,0,0]
[2,0,0,0]
Deliver!
[1,0,0,0]
Deliver this!
Deliver buffered <P1, seq:2>
Update [2,0,0,0]
P3, seq: 1
[2,0,1,0]
[2,0,1,0]
Deliver!
[2,0,1,0]
Deliver!
?
FIFO Ordering: Example
P2
Time
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
Deliver!
P1, seq: 1
[1,0,0,0]
Deliver!
[0,0,0,0]
Buffer!
P1, seq: 2
[1,0,0,0] [2,0,0,0]
[2,0,0,0]
Deliver!
[1,0,0,0]
Deliver this!
Deliver buffered <P1, seq:2>
Update [2,0,0,0]
P3, seq: 1
[2,0,1,0]
[2,0,1,0]
Deliver!
[2,0,1,0]
Deliver!
[1,0,1,0]
Deliver!
[2,0,1,0]
Deliver!
FIFO Ordering: Example
• Ensures all receivers receive all
multicasts in the same order
• Formally
– If a correct process P delivers
message m before m’
(independent of the senders),
then any other correct process
P’ that delivers m’ would already
have delivered m.
Total Ordering
24
• Special process elected as leader or sequencer
• Send multicast at process Pi:
– Send multicast message M to group and sequencer
• Sequencer:
– Maintains a global sequence number S (initially 0)
– When it receives a multicast message M, it sets S = S + 1, and
multicasts <M, S>
• Receive multicast at process Pi:
– Pi maintains a local received global sequence number Si (initially 0)
– If Pi receives a multicast M from Pj, it buffers it until it both
1. Pi receives <M, S(M)> from sequencer, and
2. Si + 1 = S(M)
• Then deliver message M to application and set Si = Si + 1
Sequencer-based Approach
25
• Multicasts whose send events are
causally related, must be received in
the same causality-obeying order at
all receivers
• Formally
– If multicast(g,m)  multicast(g,m’)
then any correct process that
delivers m’ would already have
delivered m.
– ( is Lamport’s happens-before)
Causal Ordering
26
• Each receiver maintains a vector of
per-sender sequence numbers
(integers)
– Similar to FIFO Multicast,
but updating rules are different
– Processes P1 through PN
– Pi maintains a vector Pi[1…N]
(initially all zeroes)
– Pi[j] is the latest sequence number Pi
has received from Pj
Causal Multicast: Datastructures
27
• Send multicast at process Pj:
– Set Pj[j] = Pj[j] + 1
– Include new entire vector Pj[1…N] in multicast message as its sequence number
• Receive multicast: If Pi receives a multicast from Pj with vector
M[1…N] (= Pj[1…N]) in message, buffer it until both:
1. This message is the next one Pi is expecting from Pj, i.e.,
• M[j] = Pi[j] + 1
2. All multicasts, anywhere in the group, which happened-before M have been
received at Pi, i.e.,
• For all k ≠ j: M[k] ≤ Pi[k]
• i.e., Receiver satisfies causality
3. When above two conditions satisfied, deliver M to application and set Pi[j] = M[j]
Causal Multicast: Updating Rules
28
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Causal Ordering: Example 29
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
Causal Ordering: Example
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
[1,1,0,0]
Deliver!
Missing 1 from P1
Buffer!
Causal Ordering: Example
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
[1,1,0,0]
Deliver!
Missing 1 from P1
Buffer!
[1,0,0,1]
Deliver!
Receiver satisfies causality
Deliver!
Receiver satisfies causality
Causal Ordering: Example
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
[1,1,0,0]
Deliver!
Missing 1 from P1
Buffer!
[1,0,0,1]
Deliver!
Receiver satisfies causality
Deliver!
Receiver satisfies causality
Missing 1 from P1
Buffer!
Causal Ordering: Example
Causal Ordering: Example
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
[1,1,0,0]
Deliver!
Missing 1 from P1
Buffer!
[1,0,0,1]
Deliver!
Receiver satisfies causality
Deliver!
Receiver satisfies causality
Missing 1 from P1
Buffer!
Deliver P1’s multicast
Receiver satisfies causality for buffered multicasts
Deliver P2’s buffered multicast
Deliver P4’s buffered multicast
Causal Ordering: Example
Time
P2
P1
P3
P4
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[0,0,0,0]
[1,0,0,0]
[1,0,0,0]
Deliver!
[1,0,0,0]
Deliver!
[1,1,0,0]
[1,1,0,0]
Deliver!
Missing 1 from P1
Buffer!
[1,0,0,1]
Deliver!
Receiver satisfies causality
Deliver!
Receiver satisfies causality
Missing 1 from P1
Buffer!
Deliver P1’s multicast
Receiver satisfies causality for buffered multicasts
Deliver P2’s buffered multicast
Deliver P4’s buffered multicast
Deliver!
• Ordering of multicasts affects correctness
of distributed systems using multicasts
• Three popular ways of implementing
ordering
– FIFO, Causal, Total
• And their implementations
• What about reliability of multicasts?
• What about failures?
Summary: Multicast Ordering
36
• Reliable multicast loosely says that
every process in the group receives
all multicasts
– Reliability is orthogonal to ordering
– Can implement Reliable-FIFO, or
Reliable-Causal, or Reliable-Total, or
Reliable-Hybrid protocols
• What about process failures?
• Definition becomes vague
Reliable Multicast
37
• Need all correct (i.e., non-
faulty) processes to receive
the same set of multicasts as
all other correct processes
– Faulty processes stop anyway,
so we won’t worry about them
Reliable Multicast (under failures)
38
• Let’s assume we have reliable unicast
(e.g., TCP) available to us
• First-cut: Sender process (of each multicast
M) sequentially sends a reliable unicast
message to all group recipients
• First-cut protocol does not satisfy reliability
– If sender fails, some correct processes
might receive multicast M, while other
correct processes might not receive M
Implementing Reliable Multicast
39
• Trick: Have receivers help the sender
1. Sender process (of each multicast M)
sequentially sends a reliable unicast
message to all group recipients
2. When a receiver receives multicast
M, it also sequentially sends M to all
the group’s processes
REALLY Implementing Reliable Multicast
40
• Not the most efficient multicast protocol,
but reliable
• Proof is by contradiction
• Assume two correct processes Pi and Pj are so
that Pi received a multicast M and Pj did not
receive that multicast M
– Then Pi would have sequentially sent the
multicast M to all group members, including Pj,
and Pj would have received M
– A contradiction
– Hence our initial assumption must be false
– Hence protocol preserves reliability
Analysis
41
• Attempts to preserve multicast ordering
and reliability in spite of failures
• Combines a membership protocol with a
multicast protocol
• Systems that implemented it (like Isis
Systems) have been used in NYSE, French
Air Traffic Control System, Swiss Stock
Exchange
Virtual Synchrony or View Synchrony
42
• Each process maintains a membership list
• The membership list is called a View
• An update to the membership list is called a View Change
– Process join, leave, or failure
• Virtual synchrony guarantees that all view changes are delivered in the
same order at all correct processes
– If a correct P1 process receives views, say {P1}, {P1, P2, P3}, {P1, P2}, {P1, P2, P4}
then
– Any other correct process receives the same sequence of view changes (after it joins the
group)
• P2 receives views {P1, P2, P3}, {P1, P2}, {P1, P2, P4}
• Views may be delivered at different physical times at processes, but
they are delivered in the same order
Views
43
• A multicast M is said to be “delivered in a view V at process Pi” if
– Pi receives view V, and then sometime before Pi receives the next view it
delivers multicast M
• Virtual synchrony ensures that
1. The set of multicasts delivered in a given view is the same set at all correct
processes that were in that view
• What happens in a View, stays in that View
2. The sender of the multicast message (and the send event) also belongs to that
view
3. If a process Pi does not deliver a multicast M in view V while other processes in
the view V delivered M in V, then Pi will be forcibly removed from the next view
delivered after V at the other processes
VSync Multicasts
44
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Satisfies virtual synchrony 45
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Does not satisfy virtual synchrony
Crash
46
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2}
View{P1,P2}
M1
M2
M3
Satisfies virtual synchrony
Crash
47
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Does not satisfy virtual synchrony 48
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2 (not delivered at P2)
M3
Satisfies virtual synchrony 49
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Does not satisfy virtual synchrony 50
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Does not satisfy virtual synchrony 51
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1,P2,P3}
View{P1,P2,P3}
View{P1,P2,P3}
M1
M2
M3
Satisfies virtual synchrony 52
• Again, orthogonal to virtual synchrony
• The set of multicasts delivered in a
view can be ordered either
– FIFO
– Or Causally
– Or Totally
– Or using a hybrid scheme
What about Multicast Ordering?
53
• Called “virtual synchrony” since in spite
of running on an asynchronous network,
it gives the appearance of a synchronous
network underneath that obeys the same
ordering at all processes
• So can this virtually synchronous system
be used to implement consensus?
• No! VSync groups susceptible to
partitioning
– E.g., due to inaccurate failure detections
About that name
54
Time
P2
P1
P3
P4
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
View{P1,P2,P3,P4}
Crash
View{P1}
View{P2, P3}
View{P2,P3}
M1
M2
M3
Partitioning in View synchronous systems 55
• Multicast an important building
block for cloud computing systems
• Depending on application need,
can implement
– Ordering
– Reliability
– Virtual synchrony
Summary
56
Announcements
• HW3, MP3 released. Start now!
• Midterms handed back Fri 10/11.
• Deadline for regrades: 10/18
• Final exam: 12/12 7-10PM
– Have a conflict with another final? Fill out the form on Piazza by
TODAY!
– (travel, interviews, etc., are not conflicts)
57

More Related Content

PPTX
slides.06.pptx
PDF
OS Process Chapter 3.pdf
PPTX
Chapter 3 - Operating System Scheduling.pptx
PPT
Unit2 CPU Scheduling 24252 (sssssss1).ppt
PPT
Unit2 CPU Scheduling 24252.ppBBBBBBBBBBt
PPT
3 process scheduling
slides.06.pptx
OS Process Chapter 3.pdf
Chapter 3 - Operating System Scheduling.pptx
Unit2 CPU Scheduling 24252 (sssssss1).ppt
Unit2 CPU Scheduling 24252.ppBBBBBBBBBBt
3 process scheduling

Similar to COMMUNICATION PRIMITIVES Of distributed systems .ppt (20)

PPT
PPT
fggggggggggggggggggggggggggggggfffffffffffffffffff
PPT
3_process_scheduling.ppt----------------
PPT
3_process_scheduling.ppt
PPT
Process Scheduling Algorithms for Operating Systems
PPT
3_process_scheduling.ppt
PPT
Operating System 5
PPT
OS Process and Thread Concepts
PDF
Totem
PDF
Unit iios process scheduling and synchronization
PPT
cpu sechduling
PPT
Process management in os
PPT
Operating Systems Process Scheduling Algorithms
PPTX
Presentation by adeel
PPT
Scheduling algorithms
PPT
Csc4320 chapter 5 2
PPT
Os..
PPTX
Basic Paxos Implementation in Orc
fggggggggggggggggggggggggggggggfffffffffffffffffff
3_process_scheduling.ppt----------------
3_process_scheduling.ppt
Process Scheduling Algorithms for Operating Systems
3_process_scheduling.ppt
Operating System 5
OS Process and Thread Concepts
Totem
Unit iios process scheduling and synchronization
cpu sechduling
Process management in os
Operating Systems Process Scheduling Algorithms
Presentation by adeel
Scheduling algorithms
Csc4320 chapter 5 2
Os..
Basic Paxos Implementation in Orc
Ad

Recently uploaded (20)

PDF
Operating System & Kernel Study Guide-1 - converted.pdf
PPTX
CH1 Production IntroductoryConcepts.pptx
PPTX
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
PPTX
OOP with Java - Java Introduction (Basics)
PPTX
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
PDF
Enhancing Cyber Defense Against Zero-Day Attacks using Ensemble Neural Networks
PDF
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
PDF
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
PPTX
web development for engineering and engineering
PDF
Automation-in-Manufacturing-Chapter-Introduction.pdf
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
PDF
composite construction of structures.pdf
PPTX
Current and future trends in Computer Vision.pptx
PPT
Project quality management in manufacturing
PPTX
additive manufacturing of ss316l using mig welding
PPTX
Foundation to blockchain - A guide to Blockchain Tech
PPTX
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
PPT
Introduction, IoT Design Methodology, Case Study on IoT System for Weather Mo...
PDF
Well-logging-methods_new................
PPTX
Artificial Intelligence
Operating System & Kernel Study Guide-1 - converted.pdf
CH1 Production IntroductoryConcepts.pptx
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
OOP with Java - Java Introduction (Basics)
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
Enhancing Cyber Defense Against Zero-Day Attacks using Ensemble Neural Networks
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
web development for engineering and engineering
Automation-in-Manufacturing-Chapter-Introduction.pdf
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
composite construction of structures.pdf
Current and future trends in Computer Vision.pptx
Project quality management in manufacturing
additive manufacturing of ss316l using mig welding
Foundation to blockchain - A guide to Blockchain Tech
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
Introduction, IoT Design Methodology, Case Study on IoT System for Weather Mo...
Well-logging-methods_new................
Artificial Intelligence
Ad

COMMUNICATION PRIMITIVES Of distributed systems .ppt