Multipath TCP

Decoupling TCP from IP with
Multipath TCP
Olivier Bonaventure
http://inl.info.ucl.ac.be
http://perso.uclouvain.be/olivier.bonaventure
Thanks to Sébastien Barré, Christoph Paasch, Grégory Detal, Mark
Handley, Costin Raiciu, Alan Ford, Micchio Honda, Fabien Duchene and
many others
March 2013

Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases

The origins of TCP

Source : http://spectrum.ieee.org/computing/software/the-strange-birth-and-long-life-of-unix

The Unix pipe model

1234 abbsbbbs
echo wc

The TCP bytestream model

Client Server
ABCDEF...111232

0988989 ... XYZZ

IP:1.2.3.4 IP:4.5.6.7

Endhosts have evolved

Mobile devices have multiple wireless interfaces

What technology provides
3G celltower

IP 1.2.3.4

3G celltower

IP 1.2.3.4

IP 5.6.7.8

3G celltower

IP 1.2.3.4

IP 5.6.7.8

When IP addresses change TCP connections
have to be re-established !

The Internet architecture
that we explain to our students
Application
Transport
Datalink
Network
Physical
Datalink
Physical

Network
Datalink
Physical
Physical

O. Bonaventure, Computer networking : Principles, Protocols and Practice, open ebook, http://inl.info.ucl.ac.be/cnp3

A typical "academic" network

Application Application
Transport Transport
Network Network Network
Datalink Datalink Datalink Datalink
Physical Physical Physical Physical

The end-to-end principle

Application Application
TCP
Transport Transport
Network Network Network

In reality

– almost as many middleboxes as routers
– various types of middleboxes are deployed
Sherry, Justine, et al. "Making middleboxes someone else's problem: Network processing as a cloud service."
Proceedings of the ACM SIGCOMM 2012 conference. ACM, 2012.

A middlebox zoo
Web Security VPN Concentrator
Appliance

SSL NAC Appliance
Terminator

ACE XML Cisco IOS Firewall
PIX Firewall
Gateway Right and Left

IP Telephony
Streamer Voice
Router
V Gateway

NAT
Content
Engine

http://www.cisco.com/web/about/ac50/ac47/2.html

How to model those middleboxes ?
• In the official architecture, they do not exist
• In reality...

Application Application Application
TCP
Transport Transport Transport
Network Network Network Network

TCP segments processed by a router
Ver IHL ToS Total length Ver IHL ToS Total length
Identification Flags Frag. Offset Identification Flags Frag. Offset
IP TTL Protocol Checksum TTL Protocol Checksum
Source IP address Source IP address
Destination IP address Destination IP address
Source port Destination port Source port Destination port
Sequence number Sequence number
Acknowledgment number Acknowledgment number
TCP THL Reserved Flags Window THL Reserved Flags Window
Checksum Urgent pointer Checksum Urgent pointer
Options Options

Payload Payload

TCP segments processed by a NAT
TTL Protocol Checksum TTL Protocol Checksum
THL Reserved Flags Window THL Reserved Flags Window
Options Options

Payload Payload

TCP segments processed by a NAT (2)
• active mode ftp behind a NAT
220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]
ftp_login: user `<null>' pass `<null>' host `ftp.belnet.be'
Name (ftp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connection for file list
lrw-r--r-- 1 ftp ftp 6 Jun 1 2011 pub -> mirror
226 Transfer complete

TCP segments processed by an ALG
running on a NAT
Options Options

Payload Payload

How transparent is the Internet ?
• 25th September 2010
to 30th April 2011
• 142 access networks
• 24 countries
• Sent specific TCP
segments from client
to a server in Japan

Honda, Michio, et al. "Is it still possible to extend TCP?" Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011. © O. Bonaventure, 2011

End-to-end transparency today

Middleboxes don't change
the Protocol field, but
Destination IP address
many discardaddress with an
Destination IP packets

unknown Protocol field




Options Options

Payload Payload

Design objectives
• Multipath TCP is an evolution of TCP

• Design objectives
– Support unmodified applications
– Work over today’s networks
– Works in all networks where regular TCP works

TCP Connection establishment
• Three-way handshake

SYN,seq=1234,Options

SYN+ACK,ack=1235,seq=5678,Options
ACK,seq=1235,ack=5679

Data transfer

seq=1234,"abcd"

ACK,ack=1238,win=4

seq=1238,"efgh"

ACK,ack=1242,win=0

Connection release

seq=1234,"abcd"

RST

Connection release

seq=1234,"abcd"
FIN, seq=1238
ACK,ack=1239

seq=345,"ijkl"
FIN,seq=349

FIN,ack=350

Identification of a TCP connection
Ver IHL ToS Total length Four tuple
Identification Flags Frag. Offset
IP
TTL Protocol Checksum
– IPsource
Source IP address – IPdest
Source port Destination port – Portsource
Sequence number
– Portdest
Acknowledgment number
TCP
THL Reserved Flags Window
Checksum Urgent pointer
Options
All TCP segments
contain the four
Payload
tuple

The new bytestream model

Client Server
ABCDEF...111232

0988989 ... XYZZ

D C B A
IP:2.3.4.5 IP:6.7.8.9

IP:4.5.6.7
IP:1.2.3.4

38

The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?

• Congestion control
– How to control congestion over multiple paths ?

A naïve Multipath TCP

SYN+Option
SYN+ACK+Option
ACK
seq=123, "abc"

seq=126, "def"

A naïve Multipath TCP
In today's Internet ?
SYN+Option
SYN+ACK+Option
ACK
seq=123, "abc"

There is no
corresponding
TCP connection
seq=126, "def"

Design decision

– A Multipath TCP connection is composed of one of
more regular TCP subflows that are combined

• Each host maintains state that glues the TCP subflows
that compose a Multipath TCP connection together

• Each TCP subflow is sent over a single path and appears
like a regular TCP connection along this path

Multipath TCP and the architecture

Application

socket
Application Multipath TCP
Transport
Network TCP1 TCP2 ... TCPn

Datalink
Physical

A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for multipath TCP
development", RFC6182 2011.

A regular TCP connection
• What is a regular TCP connection ?

– It starts with a three-way handshake
• SYN segments may contain special options

– All data segments are sent in sequence
• There is no gap in the sequence numbers

– It is terminated by using FIN or RST

Multipath TCP
SYN+Option
SYN+ACK+Option
ACK

SYN+OtherOption
SYN+ACK+OtherOption
ACK

How to combine two TCP subflows ?

SYN+Option
SYN+ACK+Option
ACK

SYN+OtherOption How to link with
SYN+ACK+OtherOption blue subflow ?
ACK

How to link TCP subflows ?
SYN, Portsrc=1234,Portdst=80+Option

SYN+ACK[...]
A NAT could change
ACK addresses and
port numbers

SYN, Portsrc=1235,Portdst=80
+Option[link Portsrc=1234,Portdst=80]

How to link TCP subflows ?
+Option[Token=5678]
SYN+ACK+Option[Token=6543]
ACK
MyToken=5678
YourToken=6543

MyToken=6543
YourToken=5678

+Option[Token=6543]

Subflow agility
• Multipath TCP supports
– addition of subflows
– removal of subflows

How to transfer data ?
seq=123,"a"

ack=124
seq=125,"c"
ack=126

seq=124,"b"
ack=125
seq=126,"d"
ack=127

How to transfer data
in today's Internet ?
seq=123,"a"

ack=124
seq=125,"c"
ack=126
Gap in sequence numbering space
Some DPI will not allow this !

seq=124,"b"
ack=125

Multipath TCP Data transfer
• Two levels of sequence numbers

ABCDEF

socket socket

Multipath TCP Multipath TCP
Data sequence #
TCP1 TCP1 sequence # TCP1

TCP2 TCP2 sequence # TCP2

Multipath TCP
Data transfer
Dseq=0,seq=123,"a"

DAck=1,ack=124
DSeq=2, seq=124,"c"
DAck=3, ack=125

DSeq=1, seq=456,"b"
DAck=2,ack=457

Multipath TCP
How to deal with losses ?
• Data losses over one TCP subflow
– Fast retransmit and timeout as in regular TCP
Dseq=0,seq=123,"a"
DAck=1,ack=12
4 Dseq=0,seq=123,"a"
DAck=1,ack=124

Multipath TCP
• What happens when a TCP subflow fails ?
Dseq=0,seq=123,"a"

DSeq=1, seq=456,"b"
DSeq=0,ack=457
Dseq=0,seq=457,"a"
DAck=2,ack=458

Retransmission heuristics
• Heuristics used by current Linux implementation

– Fast retransmit is performed on the same subflow
as the original transmission
– Upon timeout expiration, reevaluate whether the
segment could be retransmitted over another subflow

– Upon loss of a subflow, all the unacknowledged data
are retransmitted on other subflows

Flow control
• How should the window-based flow control
be performed ?

– Independant windows on each TCP subflow

– A single window that is shared among all TCP
subflows

Independant windows

Dseq=0,seq=123,"a"

DAck=1,ack=124,win=0

DSeq=1, seq=456,"b"
Dseq=2,seq=457,"c"

Independant windows
possible problem
Dseq=0,seq=123,"a"

DSeq=1, seq=456,"b"

• Impossible to retransmit, window is already full on green
subflow

A single window shared by all subflows

Dseq=0,seq=123,"a"


DSeq=1, seq=456,"b"
Dseq=2,seq=457,"c"

A single window shared by all subflows
Impact of middleboxes

Dseq=0,seq=123,"a"

0

DSeq=1, seq=456,"b"


Multipath TCP Windows
• Multipath TCP maintains one window per Multipath
TCP connection

– Window is relative to the last acked data (Data Ack)
– Window is shared among all subflows
• It's up to the implementation to decide how the window is shared
– Window is transmitted inside the window field of the
regular TCP header
– If middleboxes change window field,
• use largest window received at MPTCP-level
• use received window over each subflow to cope with the flow
control imposed by the middlebox

Multipath TCP buffers
MPTCP-
send(...)
level, resequen
cing possible recv(...)
socket

Multipath TCP
Scheduler

TCP1

TCP2

Transmit
queues, process Reorder
only regular TCP queue, processes
header only TCP header

Sending Multipath TCP information
• How to exchange the Multipath TCP specific
information between two hosts ?

• Option 1
– Use TLVs to encode data and control information
inside payload of subflows

• Option 2
– Use TCP options to encode all Multipath TCP
information

Option 1 : Michael Scharf, Thomas-Rolf Banniza , MCTCP: A Multipath Transport Shim Layer, GLOBECOM 2011

Is it safe to use TCP options ?
• Known option (TS) in Data segments

XD6BHM

Honda, Michio, et al. "Is it still possible to extend TCP?." Proceedings of the 2011 ACM

Is it safe to use TCP options ?
• Unknown option in Data segments

XD6BHM


Multipath TCP options
• TCP option format

Kind Length Option-specific data

• Initial design
– One option kind for each purpose
(e.g. Data Sequence number)

• Final design
– A single variable-length Multipath TCP option

Multipath TCP option
• A single option type
– to minimise the risk of having one option
accepted by middleboxes in SYN segments and
rejected in segments carrying data

Kind Length Subtype
Subtype specific data
(variable length)

Data sequence numbers
and TCP segments
• How to transport Data sequence numbers ?
– Same solution as for TCP
• Data sequence number in TCP option is the Data
sequence number of the first byte of the segment
Source port Destination port
Sequence number
Acknowledgment number
THL Reserved Flags Window
Datasequence number

Payload

TCP sequence number
and middleboxes


Which middleboxes change
TCP sequence numbers ?
• Some firewalls change TCP sequence numbers
in SYN segments to ensure randomness
– fix for old windows95 bug

• Transparent proxies terminate TCP
connections

Other types of
middlebox interference

• Data segments

Data,seq=12,"ab"

Data,seq=14,"cd"
Data,seq=12,"abcd"

Such a middlebox could also be the network adapter of the
server that uses LRO to improve performance.

Segment coalescing


and middleboxes buffers small
seq=123,Dseq=0, "a" segments
seq=124, DSeq=2,"c" seq=123, DSeq=2, "ac"

copies one option seq=123, DSeq=0, "ac"
in coalesced
segment

seq=456, DSeq=1, "b"

and middleboxes
seq=123,Dseq=0, "ab"

DSeq=0, seq=123,"a"
Middlebox only DSeq=0, seq=124,"b"
understands
regular TCP

A "middlebox" that both splits and
coalesces TCP segments

and middleboxes
• How to avoid desynchronisation between the
bytestream and data sequence numbers ?

• Solution
– Multipath TCP option carries mapping between
Data sequence numbers and (difference between
initial and current) subflow sequence numbers
• mapping covers a part of the bytestream (length)

Multipath TCP
Data transfer
seq=123,DSS[0->123,len=1],"a"

DAck=1,ack=124
seq=124, DSS[2->124,len=1],"c"
DAck=3, ack=125

seq=456, DSS[1->456,len=1],"b"
DAck=2,ack=457

and middleboxes
seq=123,DSS[0->123,len=1], "a"

seq=124, seq=123,
DSS[2->124, len=1],"c" DSS[0->123, len=1],
"ac"
DAck=2,ack=124
seq=124, DSS[2->124, len=1],"c"

seq=456, DSS[1->456, len=1], "b"
DSeq=0,ack=457

Multipath TCP and middleboxes
• With the DSS mapping, Multipath TCP can
cope with middleboxes that
– combine segments
– split segments

• Are they the most annoying middleboxes for
Multipath TCP ?

– Unfortunately not

The worst middlebox
seq=123,
DSS[1- seq=123,
>123,len=2], "ab" DSS[1->123, len=2],
"aXXXb"
DAck=3,ack=125
DAck=3,ack=128
seq=125,
DSS[3->125, len=2], seq=128,
"cd" DSS[3->125, len=2],
"cd"

• Is this an academic exercise or reality ?

The worst middlebox
• Is unfortunately very old...
– Any ALG for a NAT
220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]
ftp_login: user `<null>' pass `<null>' host `ftp.belnet.be'
Name (ftp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connection for file list
lrw-r--r-- 1 ftp ftp 6 Jun 1 2011 pub -> mirror
226 Transfer complete

Coping with the worst middlebox
• What should Multipath TCP do in the
presence of such a worst middlebox ?
– Do nothing and ignore the middlebox
• but then the bytestream and the application would be
broken and this problem will be difficult to debug by
network administrators

– Detect the presence of the middlebox
• and fallback to regular TCP (i.e. use a single path and
nothing fancy)

Multipath TCP MUST work in all networks where
regular TCP works.

Detecting the worst middlebox ?
• How can Multipath TCP detect a middlebox
that modifies the bytestream and
inserts/removes bytes ?

– Various solutions were explored

– In the end, Multipath TCP chose to include its own
checksum to detect insertion/deletion of bytes

The worst middlebox
seq=123, seq=123,
DSS[1- DSS[1->123, len=2,Inv],
>123,len=2,V], "ab" "aXXXb"

RST, last DSeq=0 RST, last DSeq=0

seq=456, DSS[1->456, len=2,V], "ab"
DAck=3,ack=458

Multipath TCP
• What should be the length of the data
sequence numbers ?

– 32 bits
• compact and compatible with TCP
• wrap around problem at highspeed requires PAWS

– 64 bits
• wrap around is not an issue for most transfers today
• takes more space inside each segment

Multipath TCP
• Data sequence numbers and Data
acknowledgements

– Maintained inside implementation as 64 bits field

– Implementations can, as an optimisation, only
transmit the lower 32 bits of the data sequence
and acknowledgements

Data Sequence Signal option
A = Data ACK present
a = Data ACK is 8 octets
Cumulative Data ack M = mapping present
m = DSN is 8

Length of mapping, can extend
Computed over data covered by
beyond this segment
entire mapping + pseudo header

Cost of the DSN checksum

C. Raiciu, et al. “How hard can it be? designing and implementing a deployable multipath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, 2012.

TCP congestion control
• A linear rate adaption algorithm

To be fair and efficient, a linear algorithm must use
additive increase and multiplicative decrease
(AIMD)
# Additive Increase Multiplicative Decrease
if congestion :
rate=rate*betaC # multiplicative decrease, betaC<1
else
rate=rate+alphaN # additive increase, v0>0

AIMD in TCP
• Congestion control mechanism
– Each host maintains a congestion window (cwnd)
– No congestion
• Congestion avoidance (additive increase)
– increase cwnd by one segment every round-trip-time
– Congestion
• TCP detects congestion by detecting losses
• Mild congestion (fast retransmit – multiplicative decrease)
– cwnd=cwnd/2 and restart congestion avoidance
• Severe congestion (timeout)
– cwnd=1, set slow-start-threshold and restart slow-start

Evolution of the congestion window

Cwnd Fast retransmit

Fast retransmit

Threshold
Threshold

Time
Slow-start Congestion avoidance
exponential increase of cwnd linear increase of cwnd

Congestion control for Multipath TCP
• Simple approach
– independant congestion windows

Threshold
Threshold

Threshold

Independant congestion windows
• Problem

12Mbps

Coupling the congestion windows
• Principle
– The TCP subflows are not independant and their
congestion windows must be coupled

• EWTCP
– For each ACK on path r, cwinr=cwinr+a/cwinr (in
segments)
– For each loss on path r, cwinr=cwinr/2

– Each subflow gets window size proportional to a2
– Same throughput as TCP if a = 1
n
M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda. Multipath Congestion Control for Shared Bottleneck. In
Proc. PFLDNeT workshop, May 2009.

Can we split traffic equally
among all subflows ?

12Mbps

12Mbps 12Mbps

In this scenario, EWTCP would get 3.5 Mbps on the two
hops path and 5 Mbps on the one hop path, less than
the optimum of 12 Mbps for each Multipath TCP connection
D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, “Design, implementation and evaluation of congestion
control for multipath TCP,” NSDI'11: Proceedings of the 8th USENIX conference on Networked systems design
and implementation, 2011.

Linked increases congestion control
• Algorithm
– For each loss on path r, cwinr=cwinr/2

– Additive increase
cwndi
max( 2
)
(rtti ) 1
cwinr = cwinr + min( , )
cwndi 2 cwndr
(å )
i rtti

D. Wischik, C. Raiciu, A. Greenhalgh, and M. Handley, “Design, implementation and evaluation of congestion
control for multipath TCP,” NSDI'11: Proceedings of the 8th USENIX conference on Networked systems design
and implementation, 2011.

Other Multipath-aware
congestion control schemes
R. Khalili, N. Gast, M. Popovic, U. Upadhyay, J.-Y. Le Boudec , MPTCP is not
Pareto-optimal: Performance issues and a possible solution, Proc. ACM
Conext 2012
Y. Cao, X. Mingwei, and X. Fu, “Delay-based Congestion Control for Multipath
TCP,” ICNP2012, 2012.
T. A. Le, C. S. Hong, and E.-N. Huh, “Coordinated TCP Westwood congestion
control for multiple paths over wireless networks,” ICOIN '12: Proceedings of the
The International Conference on Information Network 2012, 2012, pp. 92–96.
T. A. Le, H. Rim, and C. S. Hong, “A Multipath Cubic TCP Congestion Control
with Multipath Fast Recovery over High Bandwidth-Delay Product Networks,”
IEICE Transactions, 2012.
T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, “Applying TCP-Friendly
Congestion Control to Concurrent Multipath Transfer,” Advanced Information
Networking and Applications (AINA), 2010 24th IEEE International Conference on,
2010, pp. 312–319.

The Multipath TCP control plane
• Connection establishment in details

• Closing a Multipath TCP connection

• Address dynamics

Security threats
• Three main security threats were considered
– flooding attack

– man-in-the middle attack

– hijacking attach Security goal :
Multipath TCP should not be
worse than regular TCP

J. Diez, M. Bagnulo, F. Valera, and I. Vidal, “Security for multipath TCP: a constructive approach,” International
Journal of Internet Protocol Technology, vol. 6, 2011.

Multipath TCP
Connection establishment
• Principle
SYN, MP_CAPABLE
SYN+ACK, MP_CAPABLE
ACK, MP_CAPABLE

seq=123, DSeq=1, "abc"

Roles of the initial TCP handshake
• Check willingness to open TCP connection
– Propose initial sequence number
– Negotiate Maximum Segment Size
• TCP options
– negotiate Timestamps, SACK, Window scale
• Multipath TCP
– check that server supports Multipath TCP
– propose Token in each direction
– propose initial Data sequence number in each
direction
– Exchange keys to authenticate subflows

How to extend TCP ?
Theory
• TCP options were invented for this purpose
– Exemple SACK

SACK enabled
SYN+SACK Permitted

SYN+ACK SACK Permitted
SACK yes
?

How to extend TCP ?
practice
• What happens when there are middleboxes
on the path ?

SYN+SACK Permitted SACK no
SYN

SYN+ACK SYN+ACK
SACK no

TCP options
• In SYN segments

XD6BHM


How to extend TCP ?
The worst case
• What happens when there are middleboxes
on the path ?

SYN+SACK Permitted SYN+SACK Permitted

SYN+ACK SACK enabled
SACK no
SYN+ACK SACK Permitted

Client and server do not agree
on TCP extension !!!

Multipath TCP handshake

SYN, MP_CAPABLE[...]
SYN+ACK, MP_CAPABLE[...]
ACK+MPTCPOption

Why an option in
third ACK ?

Multipath TCP option in third ACK
SYN, MP_CAPABLE[...]

SYN+ACK, MP_CAPABLE[...]
SYN+ACK
ACK

No option, disable
Multipath TCP

Multipath TCP handshake
Token exchange

SYN, MP_CAPABLE[ClientToken=1234]
SYN+ACK, MP_CAPABLE[ServerToken=5678]
ACK, MP_CAPABLE[ClientToken=1234]

Useful if server
wants to send
SYN+ACK without
keeping any state

Initial Data Sequence number
• Why do we need an initial Data Sequence
number ?

– Setting IDSN to a random value improves security

– Hosts must know IDSN to avoid losing data in
some special cases

SYN, MP_CAPABLE

SYN+ACK, MP_CAPABLE
ACK
seq=123,
DSS[456->123,len=2],"ab"
First Data or not ?

SYN...
SYN+ACK...
ACK

seq=789, DSS[458->789, len=2], "cd"

• How to negotiate the IDSN ?
SYN, MP_CAPABLE[DSeq=23456]
SYN+ACK, MP_CAPABLE[DSeq=67890]
ACK

How to secure Multipath TCP
• Main goal
– Authenticate the establishment of subflows

• Principles
– Each host announces a key during initial
handshake
• keys are exchanged in clear
– When establishing a subflow, use HMAC + key to
authenticate subflow

Key exchange
SYN, [MyKey="keyABC"]
SYN+ACK, [MyKey="keyDEF"]
ACK[MyKey="keyABC", YourKey="keyDEF"]
MyKey="keyABC" MyKey="keyDEF"
YourKey="keyDEF" YourKey="keyABC"

SYN,[NonceA=123]
SYN+ACK[NonceB=456,
HMAC(123,"keyDEF")]
ACK,[HMAC(456,"keyABC")]

Putting everything inside the SYN
• How can we place inside SYN segment ?

– Initial Data Sequence Number (64 bits)

– Token (32 bits)

– Authentication Key (the longer the better)

Constraint on TCP options
Ver IHL ToS Total length
• Total length of TCP
Identification Flags Frag. Offset
TTL Protocol Checksum
header : max 64 bytes
Source IP address
– max 44 bytes for TCP
Source port Destination port
Sequence number
options
Acknowledgment number – Options length must be
THL Reserved Flags Window multiple of 4 bytes
Options

Payload

TCP options in the wild
• MSS option [4 bytes]
– Used only inside SYN segments
• Timestamp option [10 bytes] Only 20 bytes left
– Used in potentially all segments inside SYN !
• Window scale option [3 bytes]
• SACK permitted option [2 bytes]
• Selective Acknowledgements [N bytes]
– Used in data segments

http://www.iana.org/assignments/tcp-parameters/tcp-parameters.xml

The MP_CAPABLE option
crypto
A: DSN Checksum
required or not B: Extension

Used inside SYN
Only in third ACK

Initial Data Sequence Numbers
and Tokens
• Computation of initial Data Sequence Number
IDSNA=Lower64(SHA-1(KeyA))

IDSNB=Lower64(SHA-1(KeyB)) There is a small
risk of collision,
different keys
• Computation of token same token

TokenA=Upper32(SHA-1(KeyA))

TokenB=Upper32(SHA-1(KeyB))

Cost of the Multipath TCP handshake


Closing a Multipath TCP connection
• How to close a Multipath TCP connection ?
– By closing all subflows ?
RST

RST

seq=123, DSS[1->123 ...], "ab"
seq=125, DSS[5->125,DATA_FIN ...], "end"

DAck=9,ack=128
seq=128, FIN

ack=129

seq=456, DSS[3->456], "cd"
DAck=5,ack=458

• FAST Close

SYN, [...]
SYN+ACK, [...]
ACK[...]
ACK+FAST_CLOSE
RST
SYN,[...]
SYN+ACK[...]
ACK[..]

RST

Multipath TCP
Address dynamics
• How to learn the addresses of a host ?
IP=2.3.4.5

IP=3.4.5.6
IP6=2a00:1450:400c:c05::69

• How to deal with address changes ?
IP=1.2.3.4

IP=4.5.6.7

Address dynamics
• Basic solution : multihomed server
SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR[3.4.5.6] IP=2.3.4.5
IP=3.4.5.6
ADD_ADDR[2a00:1450:400c:c05::69]
IP6=2a00:1450:400c:c05::69
SYN,[...]
SYN+ACK[...]
ACK[..]

Address dynamics
• Basic solution : mobile client
SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [4.5.6.7]
IP=2.3.4.5
SYN,[...]
IP=1.2.3.4
SYN+ACK[...]
IP=4.5.6.7
ACK[..]

REMOVE_ADDR[1.2.3.4]

Address dynamics
in today's Internet

SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [10.0.0.2]
ADD_ADDR [10.0.0.2]
IP=2.3.4.5
IP=1.2.3.4
SYN [...]
IP=10.0.0.2

Address dynamics with NATs
• Solution

– Each address has one identifier
• Subflow is established between id=0 addresses
– Each host maintains a list of <address,id> pairs of
the addresses associated to an MPTCP endpoint
– MPTCP options refer to the address identifier
• ADD_ADDR contains <address,id>
• REMOVE_ADDR contains <id>

Address dynamics

SYN, [...]

SYN+ACK, [...]
ACK[...]

ADD_ADDR [4.5.6.7,id=1]
IP=2.3.4.5
SYN,[id=1...]
IP=1.2.3.4
SYN+ACK[...]
IP=4.5.6.7
ACK[..]

REMOVE_ADDR[id=0]

Agenda
• The motivations for Multipath TCP

• The changing Internet

• The Multipath TCP Protocol

• Multipath TCP use cases
– Datacenters
– Smartphones

Datacenters evolve
• Traditional Topologies are tree-
based
– Poor performance
– Not fault tolerant …

• Shift towards multipath
topologies:
FatTree, BCube, VL2,
Cisco, EC2

C. Raiciu, et al. “Improving datacenter performance and robustness with multipath TCP,” ACM
SIGCOMM 2011.

Fat Tree Topology [Fares et al., 2008;
K=4 1953]
Clos,

Aggregation
Switches

K Pods with
K Switches
each

Racks of
servers
SIGCOMM 2011.

TCP in FAT tree networks
Cost of collissions

SIGCOMM 2011.

How to get rid of these collisions ?
• Consider TCP performance as an optimisation
problem

The Multipath TCP way
ECMP balances
the subflows
over different
paths

Two subflows
differ by their
source SIGCOMM 2011.
port

MPTCP better utilizes the FatTree network

SIGCOMM 2011.

Motivation
• One device, many IP-enabled interfaces

MPTCP over WiFi/3G

8Mbps, 20ms

2Mbps, 150ms

TCP over WiFi/3G


MPTCP over WiFi/3G


MPTCP over WiFi/3G

Multipath TCP
increases
throughput

MPTCP over WiFi/3G

What
happened
here?

Understanding the performance issue
Window full !
D C B No new data can be sent on WiFi path

8Mbps, 20ms

A 2Mbps, 150ms Window

Reinject segment on fast path
Halve congestion window on slow subflow

Usage of 3G and WiFI
• How should Multipath TCP use 3G and WiFi ?

– Full mode
• Both wireless networks are used at the same time

– Backup mode
• Prefer WiFi when available, open subflows on 3G and use
them as backup

– Single path mode
• Only one path is used at a time, WiFi preferred over 3G

Multipath TCP : Full mode

SYN...
SYN+ACK...
ACK...

WiFi

3G/LTE
SYN...

SYN+ACK,...

ACK ...

Multipath TCP : Backup mode

SYN...
SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN[Backup...]
Dynamically
SYN+ACK,...
change backup
ACK ... status of flow
MP_PRIO[B]

Multipath TCP : Backup mode
• What happens when link fails ?
SYN...
SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN[Backup...]

SYN+ACK,...
ACK ...
REM_ADDR[id=0]

Multipath TCP : single-path mode
• Multipath TCP supports break before make
SYN...
SYN+ACK...
ACK...

WiFi

3G/LTE
SYN,MP_JOIN

SYN+ACK,...
ACK ...
REM_ADDR[id=0]

Evaluation scenario
WiFi:
Belgacom ADSL2+
(~8 Mbps, ~30 ms)

3G: Mobistar
(~2 Mbps, ~80ms)

Recovery after failure

C. Paasch, et al. , “Exploring mobile/WiFi handover with multipath TCP,” presented at the CellNet '12: Proceedings
of the 2012 ACM SIGCOMM workshop on Cellular networks: operations, challenges, and future design, 2012.

Conclusion
• Multipath TCP is becoming a reality
– Due to the middleboxes, the protocol is more
complex than initially expected
– RFC has been published
– there is running code !
– Multipath TCP works over today's Internet !
• What's next ?
– More use cases
• IPv4/IPv6, anycast, load balancing, deployment
– Measurements and improvements to the protocol
• Time to revisit 20+ years of heuristics added to TCP

References
• The Multipath TCP protocol
– http://www.multipath-tcp.org
– http://tools.ietf.org/wg/mptcp/
A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for
multipath TCP development", RFC6182 2011.

A. Ford, C. Raiciu, M. J. Handley, and O. Bonaventure, “TCP Extensions for
Multipath Operation with Multiple Addresses,” RFC6824, 2013

C. Raiciu, C. Paasch, S. Barre, A. Ford, M. Honda, F. Duchene, O. Bonaventure, and
M. Handley, “How hard can it be? designing and implementing a deployable
multipath TCP,” NSDI'12: Proceedings of the 9th USENIX conference on Networked
Systems Design and Implementation, 2012.

Implementations
• Linux
– http://www.multipath-tcp.org
S. Barre, C. Paasch, and O. Bonaventure, “Multipath tcp: From theory
to practice,” NETWORKING 2011, 2011.
Sébastien Barré. Implementation and assessment of Modern Host-
based Multipath Solutions. PhD thesis. UCL, 2011
• FreeBSD
– http://caia.swin.edu.au/urp/newtcp/mptcp/
• Simulators
– http://nrg.cs.ucl.ac.uk/mptcp/implementation.html
– http://code.google.com/p/mptcp-ns3/

Middleboxes

M. Honda, Y. Nishida, C. Raiciu, A. Greenhalgh, M. Handley, and H.
Tokuda, “Is it still possible to extend TCP?,” IMC '11: Proceedings
of the 2011 ACM SIGCOMM conference on Internet measurement
conference, 2011.

V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi, “Design and
implementation of a consolidated middlebox architecture,” USENIX
NSDI, 2012.

J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V.
Sekar, “Making middleboxes someone else's problem: network
processing as a cloud service,” SIGCOMM '12: Proceedings of the ACM
SIGCOMM 2012 conference on
Applications, technologies, architectures, and protocols for computer
communication, 2012.

Multipath congestion control
– Background
D. Wischik, M. Handley, and M. B. Braun, “The resource pooling principle,”
ACM SIGCOMM Computer …, vol. 38, no. 5, 2008.
F. Kelly and T. Voice. Stability of end-to-end algorithms for joint
routing and rate control. ACM SIGCOMM CCR, 35, 2005.
P. Key, L. Massoulie, and P. D. Towsley, “Path Selection and Multipath
Congestion Control,” INFOCOM 2007. 2007, pp. 143–151.
– Coupled congestion control
C. Raiciu, M. J. Handley, and D. Wischik, “Coupled Congestion Control for
Multipath Transport Protocols,” RFC, vol. 6356, Oct. 2011.
D. Wischik, C. Raiciu, A. Greenhalgh, and M.
Handley, “Design, implementation and evaluation of congestion control
for multipath TCP,” NSDI'11: Proceedings of the 8th USENIX conference
on Networked systems design and implementation, 2011.

Multipath congestion control
– More
R. Khalili, N. Gast, M. Popovic, U. Upadhyay, J.-Y. Le Boudec , MPTCP is not
Pareto-optimal: Performance issues and a possible solution, Proc. ACM
Conext 2012
Y. Cao, X. Mingwei, and X. Fu, “Delay-based Congestion Control for Multipath
TCP,” ICNP2012, 2012.
T. A. Le, C. S. Hong, and E.-N. Huh, “Coordinated TCP Westwood congestion
control for multiple paths over wireless networks,” ICOIN '12: Proceedings of the
The International Conference on Information Network 2012, 2012, pp. 92–96.
T. A. Le, H. Rim, and C. S. Hong, “A Multipath Cubic TCP Congestion Control
with Multipath Fast Recovery over High Bandwidth-Delay Product Networks,”
IEICE Transactions, 2012.
T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, “Applying TCP-Friendly
Congestion Control to Concurrent Multipath Transfer,” Advanced Information
Networking and Applications (AINA), 2010 24th IEEE International Conference
on, 2010, pp. 312–319.

Use cases
– Datacenter
C. Raiciu, S. Barre, C. Pluntke, A. Greenhalgh, D. Wischik, and M. J.
Handley, “Improving datacenter performance and robustness with
multipath TCP,” ACM SIGCOMM 2011.
G. Detal, Ch. Paasch, S. van der Linden, P. Mérindol, G. Avoine, O.
Bonaventure, Revisiting Flow-Based Load Balancing: Stateless Path Selection in
Data Center Networks, to appear in Computer Networks
– Mobile
C. Pluntke, L. Eggert, and N. Kiukkonen, “Saving mobile device energy with
multipath TCP,” MobiArch '11: Proceedings of the sixth international
workshop on MobiArch, 2011.
C. Paasch, G. Detal, F. Duchene, C. Raiciu, and O. Bonaventure, “Exploring
mobile/WiFi handover with multipath TCP,” CellNet '12: Proceedings of
the 2012 ACM SIGCOMM workshop on Cellular networks:
operations, challenges, and future design, 2012.

Multipath TCP

More Related Content

What's hot (20)

Similar to Multipath TCP (20)

More from Olivier Bonaventure (20)

Recently uploaded (20)

Multipath TCP

Editor's Notes