4) sizing router buffers

Sizing Router Buffers
Stanford HPNG Technical Report TR04-HPNG-06-08-00

Guido Appenzeller Isaac Keslassy Nick McKeown
Stanford University Stanford University Stanford University
appenz@cs.stanford.edu keslassy@yuba.stanford.edu nickm@stanford.edu

ABSTRACT General Terms
All Internet routers contain buffers to hold packets during Design, Performance.
times of congestion. Today, the size of the buffers is deter-
mined by the dynamics of TCP’s congestion control algo- Keywords
rithm. In particular, the goal is to make sure that when a
link is congested, it is busy 100% of the time; which is equiv- Internet router, buffer size, bandwidth delay product, TCP.
alent to making sure its buffer never goes empty. A widely
used rule-of-thumb states that each link needs a buffer of 1. INTRODUCTION AND MOTIVATION
size B = RT T × C, where RT T is the average round-trip
time of a flow passing across the link, and C is the data rate 1.1 Background
of the link. For example, a 10Gb/s router linecard needs
Internet routers are packet switches, and therefore buffer
approximately 250ms × 10Gb/s = 2.5Gbits of buffers; and
packets during times of congestion. Arguably, router buffers
the amount of buffering grows linearly with the line-rate.
are the single biggest contributor to uncertainty in the Inter-
Such large buffers are challenging for router manufacturers,
net. Buffers cause queueing delay and delay-variance; when
who must use large, slow, off-chip DRAMs. And queueing
they overflow they cause packet loss, and when they under-
delays can be long, have high variance, and may destabilize
flow they can degrade throughput. Given the significance
the congestion control algorithms. In this paper we argue
of their role, we might reasonably expect the dynamics and
that the rule-of-thumb (B = RT T × C) is now outdated and
sizing of router buffers to be well understood, based on a
incorrect for backbone routers. This is because of the large
well-grounded theory, and supported by extensive simula-
number of flows (TCP connections) multiplexed together on
tion and experimentation. This is not so.
a single backbone link. Using theory, simulation and exper-
Router buffers are sized today based on a rule-of-thumb
iments on a network of real routers, we show that√ link a
commonly attributed to a 1994 paper by Villamizar and
with n flows requires no more than B = (RT T × C)/ n, for
Song [1].Using experimental measurements of at most eight
long-lived or short-lived TCP flows. The consequences on
TCP flows on a 40 Mb/s link, they concluded that — be-
router design are enormous: A 2.5Gb/s link carrying 10,000
cause of the dynamics of TCP’s congestion control algo-
flows could reduce its buffers by 99% with negligible dif-
rithms — a router needs an amount of buffering equal to
ference in throughput; and a 10Gb/s link carrying 50,000
the average round-trip time of a flow that passes through
flows requires only 10Mbits of buffering, which can easily be
the router, multiplied by the capacity of the router’s net-
implemented using fast, on-chip SRAM.
work interfaces. This is the well-known B = RT T × C rule.
We will show later that the rule-of-thumb does indeed make
Categories and Subject Descriptors sense for one (or a small number of) long-lived TCP flows.
C.2 [Internetworking]: Routers Network operators follow the rule-of-thumb and require
that router manufacturers provide 250ms (or more) of buffer-
∗The authors are with the Computer Systems Laboratory at ing [2]. The rule is found in architectural guidelines [3], too.
Stanford University. Isaac Keslassy is now with the Tech- Requiring such large buffers complicates router design, and
nion (Israel Institute of Technology), Haifa, Israel. This is an impediment to building routers with larger capacity.
work was funded in part by the Stanford Networking Re-
search Center, the Stanford Center for Integrated Systems, For example, a 10Gb/s router linecard needs approximately
and a Wakerly Stanford Graduate Fellowship. 250ms × 10Gb/s= 2.5Gbits of buffers, and the amount of
buffering grows linearly with the line-rate.
The goal of our work is to find out if the rule-of-thumb still
holds. While it is still true that most traffic uses TCP, the
number of flows has increased significantly. Today, backbone
links commonly operate at 2.5Gb/s or 10Gb/s, and carry
over 10,000 flows [4].
Technical Report TR04-HPNG-06-08-00 One thing is for sure: It is not well understood how much
buffering is actually needed, or how buffer size affects net-
This is an extended version of the publication at SIGCOMM 04
The electronic version of this report is available on the Stanford High work performance [5]. In this paper we argue that the rule-
Performance Networking group home page http://yuba.stanford.edu/ of-thumb is outdated and incorrect. We believe that sig-

20
Window [pkts]
W = 2 Tp * C
10

0
2 2.5 3 3.5 4 4.5
8
Q [pkts]
6
4
2
0
2 2.5 3 3.5 4 4.5

Figure 1: Window (top) and router queue (bottom) for a TCP flow through a bottleneck link.

link is a bottleneck, then we want to keep it busy as much
of the time as possible, and so we just need to make sure
the buffer never underflows and goes empty.
Fact: The rule-of-thumb is the amount of buffering
needed by a single TCP flow, so that the buffer at the bot-
tleneck link never underflows, and so the router doesn’t lose
Figure 2: Single flow topology consisting of an ac- throughput.
cess link of latency lAcc and link capacity CAcc and a The rule-of-thumb comes from the dynamics of TCP’s
bottleneck link of capacity C and latency l. congestion control algorithm. In particular, a single TCP
flow passing through a bottleneck link requires a buffer size
equal to the bandwidth-delay product in order to prevent the
nificantly smaller buffers could be used in backbone routers link from going idle and thereby losing throughput. Here,
(e.g. by removing 99% of the buffers) without a loss in net- we will give a quick intuitive explanation of where the rule-
work utilization, and we show theory, simulations and exper- of-thumb comes from; in particular, why this is just the right
iments to support our argument. At the very least, we be- amount of buffering if the router carried just one long-lived
lieve that the possibility of using much smaller buffers war- TCP flow. In Section 2 we will give a more precise explana-
rants further exploration and study, with more comprehen- tion, which will set the stage for a theory for buffer sizing
sive experiments needed on real backbone networks. This with one flow, or with multiple long- and short-lived flows.
paper isn’t the last word, and our goal is to persuade one Later, we will confirm that our theory is true using simula-
or more network operators to try reduced router buffers in tion and experiments in Sections 5.1 and 5.2 respectively.
their backbone network. Consider the simple topology in Figure 2 in which a single
It is worth asking why we care to accurately size router TCP source sends an infinite amount of data with packets
buffers; with declining memory prices, why not just over- of constant size. The flow passes through a single router,
buffer routers? We believe overbuffering is a bad idea for and the sender’s access link is much faster than the re-
two reasons. First, it complicates the design of high-speed ceiver’s bottleneck link of capacity C, causing packets to be
routers, leading to higher power consumption, more board queued at the router. The propagation time between sender
space, and lower density. Second, overbuffering increases and receiver (and vice versa) is denoted by Tp . Assume
end-to-end delay in the presence of congestion. Large buffers that the TCP flow has settled into the additive-increase and
conflict with the low-latency needs of real time applications multiplicative-decrease (AIMD) congestion avoidance mode.
(e.g. video games, and device control). In some cases large The sender transmits a packet each time it receives an
delays can make congestion control algorithms unstable [6] ACK, and gradually increases the number of outstanding
and applications unusable. packets (the window size), which causes the buffer to grad-
ually fill up. Eventually a packet is dropped, and the sender
1.2 Where does the rule-of-thumb come from? doesn’t receive an ACK. It halves the window size and
The rule-of-thumb comes from a desire to keep a congested pauses.1 The sender now has too many packets outstanding
link as busy as possible, so as to maximize the throughput in the network: it sent an amount equal to the old win-
of the network. We are used to thinking of sizing queues so dow, but now the window size has halved. It must therefore
as to prevent them from overflowing and losing packets. But pause while it waits for ACKs to arrive before it can resume
TCP’s “sawtooth” congestion control algorithm is designed transmitting.
to fill any buffer, and deliberately causes occasional loss to The key to sizing the buffer is to make sure that while
provide feedback to the sender. No matter how big we make the sender pauses, the router buffer doesn’t go empty and
the buffers at a bottleneck link, TCP will cause the buffer force the bottleneck link to go idle. By determining the rate
to overflow. at which the buffer drains, we can determine the size of the
Router buffers are sized so that when TCP flows pass reservoir needed to prevent it from going empty. It turns
through them, they don’t underflow and lose throughput; out that this is equal to the distance (in bytes) between the
and this is where the rule-of-thumb comes from. The metric peak and trough of the “sawtooth” representing the TCP
we will use is throughput, and our goal is to determine the
size of the buffer so as to maximize throughput of a bottle- 1
We assume the reader is familiar with the dynamics of TCP.
neck link. The basic idea is that when a router has packets A brief reminder of the salient features can be found in Ap-
buffered, its outgoing link is always busy. If the outgoing pendix A.

window size. We will show later that this corresponds to
the rule-of-thumb: B = RT T × C.
It is worth asking if the TCP sawtooth is the only factor
that determines the buffer size. For example, doesn’t sta-
tistical multiplexing, and the sudden arrival of short bursts
have an effect? In particular, we might expect the (very
bursty) TCP slow-start phase to increase queue occupancy
and frequently fill the buffer. Figure 1 illustrates the ef-
fect of bursts on the queue size for a typical single TCP Figure 3: Schematic evolution of a router buffer for
flow. Clearly the queue is absorbing very short term bursts a single TCP flow.
in the slow-start phase, while it is accommodating a slowly
changing window size in the congestion avoidance phase. We
Window [Pkts]
will examine the effect of burstiness caused by short-flows in 250

Section 4. We’ll find that the short-flows play a very small 200

effect, and that the buffer size is, in fact, dictated by the 150

number of long flows. 100
50
1.3 How buffer size influences router design 0
0 10 20 30 40 50 60 70 80 90
Having seen where the rule-of-thumb comes from, let’s see Queue [Pkts]
100
why it matters; in particular, how it complicates the design
50
of routers. At the time of writing, a state of the art router
0
linecard runs at an aggregate rate of 40Gb/s (with one or 0 10 20 30 40 50 60 70 80 90
more physical interfaces), has about 250ms of buffering, and Figure 4: A TCP flow through a single router with
so has 10Gbits (1.25Gbytes) of buffer memory. buffers equal to the delay-bandwidth product. The
Buffers in backbone routers are built from commercial upper graph shows the time evolution of the conges-
memory devices such as dynamic RAM (DRAM) or static tion window W (t). The lower graph shows the time
RAM (SRAM).2 The largest commercial SRAM chip today evolution of the queue length Q(t).
is 36Mbits, which means a 40Gb/s linecard would require
over 300 chips, making the board too large, too expensive
and too hot. If instead we try to build the linecard using this small might make little or no difference to the utilization
DRAM, we would just need 10 devices. This is because of backbone links.
DRAM devices are available up to 1Gbit. But the prob-
lem is that DRAM has a random access time of about 50ns,
which is hard to use when a minimum length (40byte) packet
2. BUFFER SIZE FOR A SINGLE LONG-
can arrive and depart every 8ns. Worse still, DRAM access LIVED TCP FLOW
times fall by only 7% per year, and so the problem is going In the next two sections we will determine how large the
to get worse as line-rates increase in the future. router buffers need to be if all the TCP flows are long-lived.
In practice router linecards use multiple DRAM chips We will start by examining a single long-lived flow, and then
in parallel to obtain the aggregate data-rate (or memory- consider what happens when many flows are multiplexed
bandwidth) they need. Packets are either scattered across together.
memories in an ad-hoc statistical manner, or use an SRAM Starting with a single flow, consider again the topology in
cache with a refresh algorithm [7]. Either way, such a large Figure 2 with a single sender and one bottleneck link. The
packet buffer has a number of disadvantages: it uses a very schematic evolution of the router’s queue (when the source
wide DRAM bus (hundreds or thousands of signals), with is in congestion avoidance) is shown in Figure 3. From time
a huge number of fast data pins (network processors and t0 , the sender steadily increases its window-size and fills the
packet processor ASICs frequently have more than 2,000 buffer, until the buffer has to drop the first packet. Just
pins making the chips large and expensive). Such wide buses under one round-trip time later, the sender times-out at
consume large amounts of board space, and the fast data time t1 , because it is waiting for an ACK for the dropped
pins on modern DRAMs consume a lot of power. packet. It immediately halves its window size from Wmax to
In summary, it is extremely difficult to build packet buffers Wmax /2 packets3 . Now, the window size limits the number
at 40Gb/s and beyond. Given how slowly memory speeds of unacknowledged (i.e. outstanding) packets in the net-
improve, this problem is going to get worse over time. work. Before the loss, the sender is allowed to have Wmax
Substantial benefits could be gained by placing the buffer outstanding packets; but after the timeout, it is only allowed
memory directly on the chip that processes the packets (a to have Wmax /2 outstanding packets. Thus, the sender has
network processor or an ASIC). In this case, very wide and too many outstanding packets, and it must pause while it
fast access to a single memory is possible. Commercial waits for the ACKs for Wmax /2 packets. Our goal is to
packet processor ASICs have been built with 256Mbits of make sure the router buffer never goes empty in order to
“embedded” DRAM. If memories of 2% the delay-bandwidth keep the router fully utilized. Therefore, the buffer must
product were acceptable (i.e. 98% smaller than they are to- not go empty while the sender is pausing.
day), then a single-chip packet processor would need no ex- If the buffer never goes empty, the router must be sending
ternal memories. We will present evidence later that buffers packets onto the bottleneck link at constant rate C. This in
2 3
DRAM includes devices with specialized I/O, such as DDR- While TCP measures window size in bytes, we will count
SDRAM, RDRAM, RLDRAM and FCRAM. window size in packets for simplicity of presentation.

250 The window size follows the familiar sawtooth pattern, in-
Window [Pkts]
200 creasing steadily until a loss occurs and then halving the win-
150 dow size before starting to increase steadily again. Notice
100
that the buffer occupancy almost hits zero once per packet
loss, but never stays empty. This is the behavior we want
50
for the bottleneck link to stay busy.
0
0 10 20 30 40 50 60 70 80 90 Appendix B presents an analytical fluid model that pro-
100
Queue [Pkts]
vides a closed-form equation of the sawtooth, and closely
50 matches the ns2 simulations.
Figures 5 and 6 show what happens if the link is under-
0
0 10 20 30 40 50 60 70 80 90 buffered or overbuffered. In Figure 5, the router is under-
Figure 5: A TCP flow through an underbuffered buffered, and the buffer size is less than RT T × C. The
router. congestion window follows the same sawtooth pattern as in
the sufficiently buffered case. However, when the window
is halved and the sender pauses waiting for ACKs, there is
350 Window [Pkts]
300 insufficient reserve in the buffer to keep the bottleneck link
250 busy. The buffer goes empty, the bottleneck link goes idle,
200 and we lose throughput.
150
100
On the other hand, Figure 6 shows a flow which is over-
50 buffered. It behaves like a correctly buffered flow in that it
0 fully utilizes the link. However, when the window is halved,
0 10 20 30 40 50 60 70 80 90
the buffer does not completely empty. The queueing delay
150 Queue [Pkts]
100 of the flows is increased by a constant, because the buffer
50 always has packets queued.
0 In summary, if B ≥ 2Tp ×C = RT T ×C, the router buffer
0 10 20 30 40 50 60 70 80 90
(just) never goes empty, and the bottleneck link will never
Figure 6: A TCP flow through an overbuffered
go idle.
router.

3. WHEN MANY LONG TCP FLOWS
turn means that ACKs arrive to the sender at rate C. The
sender therefore pauses for exactly (Wmax /2)/C seconds for SHARE A LINK
the Wmax /2 packets to be acknowledged. It then resumes In a backbone router many flows share the bottleneck link
sending, and starts increasing its window size again. simultaneously, and so the single long-lived flow is not a real-
The key to sizing the buffer is to make sure that the buffer istic model. For example, a 2.5Gb/s (OC48c) link typically
is large enough, so that while the sender pauses, the buffer carries over 10,000 flows at a time [4].4 So how should we
doesn’t go empty. When the sender first pauses at t1 , the change our model to reflect the buffers required for a bot-
buffer is full, and so it drains over a period B/C until t2 tleneck link with many flows? We will consider two situa-
(shown in Figure 3). The buffer will just avoid going empty tions. First, we will consider the case when all the flows are
if the first packet from the sender shows up at the buffer synchronized with each other, and their sawtooths march
just as it hits empty, i.e. (Wmax /2)/C ≤ B/C, or in lockstep perfectly in-phase. Then we will consider flows
that are not synchronized with each other, or are at least
B ≥ Wmax /2. not so synchronized as to be marching in lockstep. When
To determine Wmax , we consider the situation after the they are sufficiently desynchronized — and we will argue
sender has resumed transmission. The window size is now that this is the case in practice — the amount of buffering
Wmax /2, and the buffer is empty. The sender has to send drops sharply.
packets at rate C or the link will be underutilized. It is well
known that the sending rate of TCP is R = W/RT T (see 3.1 Synchronized Long Flows
equation 7 in AppendixA). Since the buffer is empty, we Consider the evolution of two TCP Reno flows through a
have no queueing delay. Therefore, to send at rate C we bottleneck router. The evolution of the window sizes, send-
require that ing rates and queue length is shown in Figure 7.
Although the two flows start at different times, they
W Wmax /2
C= = quickly synchronize to be perfectly in phase. This is a well-
RT T 2TP documented and studied tendency of flows sharing a bottle-
or Wmax /2 = 2TP × C which for the buffer leads to the neck to become synchronized over time [9, 11, 12, 13].
well-known rule-of-thumb A set of precisely synchronized flows has the same buffer
requirements as a single flow. Their aggregate behavior is
B ≥ 2Tp × C = RT T × C.
still a sawtooth; as before, the height of the sawtooth is
While not widely known, similar arguments have been made 4
previously [8, 9], and our result can be easily verified using This shouldn’t be surprising: A typical user today is con-
ns2 [10] simulation and a closed-form analytical model (Ap- nected via a 56kb/s modem, and a fully utilized 2.5Gb/s can
simultaneously carry over 40,000 such flows. When it’s not
pendix B). Figure 4 illustrates the evolution of a single TCP fully utilized, the buffers are barely used, and the link isn’t
Reno flow, using the topology shown in Figure 2. The buffer a bottleneck. So we should size the buffers for when there
size is exactly equal to the rule-of-thumb, B = RT T × C. are a large number of flows.

0.035
PDF of Aggregate Window
3500 Normal Distribution N(11000,400)
RTT [ms*10] 0.03
3000 Win [Pkts*10] Buffer = 1000 pkts
Util [Pkts/s]
Rate [Pkts/s]
2500 0.025

2000

Probability
0.02
1500 Q=0 Q>B
link underutilized packets dropped
1000 0.015

500
0.01
0
0 10 20 30 40 50 60 70 80 90
0.005
3500
RTT [ms*10]
3000 Win [Pkts*10] 0
Util [Pkts/s] 9500 10000 10500 11000 11500 12000 12500
Rate [Pkts/s]
2500 Packets

2000

1500
Figure 8: The probability distribution of the sum of
1000 the congestion windows of all flows passing through
500 a router and its approximation with a normal distri-
bution. The two vertical marks mark the boundaries
0
0 10 20 30 40 50 60 70 80 90 of where the number of outstanding packets fit into
160
140 Queue [Pkts]
the buffer. If sum of congestion windows is lower
120 and there are less packets outstanding, the link will
100
80
60 be underutilized. If it is higher the buffer overflows
40
20
and packets are dropped.
0
0 10 20 30 40 50 60 70 80 90

Solving for B we find once again that B = 2T p ·C = RT T ·C.
Figure 7: Two TCP flows sharing a bottleneck link. Clearly this result holds for any number of synchronized in-
The upper two graphs show the time evolution of phase flows.
the RTT of the flows, the congestion window of the
senders, the link utilization and the sending rate 3.2 Desynchronized Long Flows
of the TCP senders. The bottom graph shows the Flows are not synchronized in a backbone router carrying
queue length of the router buffer. thousands of flows with varying RTTs. Small variations in
RTT or processing time are sufficient to prevent synchro-
nization [14]; and the absence of synchronization has been
dictated by the maximum window size needed to fill the demonstrated in real networks [4, 15]. Likewise, we found
round-trip path, which is independent of the number of in our simulations and experiments that while in-phase syn-
flows. Specifically, assume that there are n flows, each with chronization is common for under 100 concurrent flows, it is
a congestion window W i (t) at time t, and end-to-end prop- very rare above 500 concurrent flows. 5 Although we don’t
i
agation delay Tp , where i = [1, ..., n]. The window size is precisely understand when and why synchronization of TCP
the maximum allowable number of outstanding bytes, so flows takes place, we have observed that for aggregates of
n
X over 500 flows, the amount of in-phase synchronization de-
W i (t) = 2T p · C + Q(t) (1) creases. Under such circumstances we can treat flows as
i=1 being not synchronized at all.
To understand the difference between adding synchronized
where Q(t) is the buffer occupancy at time t, and T P is the and desynchronized window size processes, recall that if we
average propagation delay. As before, we can solve for the add together many synchronized sawtooths, we get a sin-
buffer size by considering two cases: just before and just gle large sawtooth, and the buffer size requirement doesn’t
after packets are dropped. First, because they move in lock- change. If on the other hand the sawtooths are not syn-
step, the flows all have their largest window size, Wmax at chronized, the more flows we add, the less their sum will
the same time; this is when the buffer is full, so: look like a sawtooth; they will smooth each other out, and
n
X the distance from the peak to the trough of the aggregate
i
Wmax = Wmax = 2T p · C + B. (2) window size will get smaller. Hence, given that we need as
i=1 much buffer as the distance from the peak to the trough
Similarly, their window size is smallest just after they all of the aggregate window size, we can expect the buffer size
drop simultaneously [9]. If the buffer is sized so that it just 5
Some out-of-phase synchronization (where flows are syn-
goes empty as the senders start transmitting after the pause, chronized but scale down their window at different times
then during a cycle) was visible in some ns2 simulations with up
n to 1000 flows. However, the buffer requirements are very
X i Wmax similar for out-of-phase synchronization as they are for no
Wmin = = 2T p · C. (3)
i=1
2 synchronization at all.

Sum of TCP Windows [pkts]
modelled as oscillating with a uniform distribution around
2
12000 Router Queue [pkts] its average congestion window size W i , with minimum 3 W i
4
and maximum 3 W i . Since the standard deviation of the
11500
uniform distribution is √1 -th of its length, the standard
12
deviation of a single window size σWi is thus
Buffer B

„ «
1 4 2 1
σWi = √ Wi − Wi = √ Wi
12 3 3 3 3

10500 From Equation (5),

W 2T p · C + Q 2T p · C + B
Wi = = ≤ .
10000 n n n

50 52 54 56 58 60 62 64
For a large number of flows, the standard deviation of the
time [seconds] sum of the windows, W , is given by
√
σW ≤ nσWi ,
P
Figure 9: Plot of Wi (t) of all TCP flows, and of and so by Equation (5) the standard deviation of Q(t) is
the queue Q offset by 10500 packets.
1 2T p · C + B
σQ = σ W ≤ √ √ .
3 3 n
requirements to get smaller as we increase the number of
flows. This is indeed the case, and we will explain why, and Now that we know the distribution of the queue occu-
then demonstrate via simulation. pancy, we can approximate the link utilization for a given
Consider a set of TCP flows with random (and indepen- buffer size. Whenever the queue size is below a threshold, b,
dent) start times and propagation delays. We’ll assume that there is a risk (but not guaranteed) that the queue will go
they are desynchronized enough that the window size pro- empty, and we will lose link utilization. If we know the prob-
cesses are independent of each other. We can model the ability that Q < b, then we have an upper bound on the lost
total window size as a bounded random process made up of utilization. Because Q has a normal distribution, we can use
the sum of these independent sawtooths. We know from the the error-function 6 to evaluate this probability. Therefore,
central limit theorem that the aggregate window size process we get the following lower bound for the utilization.
will converge to a gaussian process. Figure 8 shows that in-
√
0 1
deed the aggregate window size does converge to a gaussian 3 3 B
process. The graph shows the probability distribution of the U til ≥ erf @ √ A. (6)
sum of the congestion windows of all flows W =
P
Wi , with 2 2 2T p√·C+B
n
different propagation times and start times as explained in
Section 5.1. Here are some numerical examples of utilization, using n =
From the window size process, we know that the queue 10000.
occupancy at time t is Router Buffer Size Utilization
n 2T p ·C
X B = 1 · √n Util ≥ 98.99 %
Q(t) = Wi (t) − (2TP · C) − . (4) 2T p ·C
i=1 B = 1.5 · √n Util ≥ 99.99988 %
2T p ·C
In other words, all outstanding bytes are in the queue (Q(t)), B = 2 · √n Util ≥ 99.99997 %
on the link (2Tp · C), or have been dropped. We represent
the number of dropped packets by . If the buffer is large This means that we can achieve full utilization with buffers
enough and TCP is operating correctly, then is negligible that are the delay-bandwidth product divided by square-
compared to 2TP · C. Therefore, the distribution of Q(t) is root of the number of flows, or a small multiple thereof. As
shown in Figure 9, and is given by the number of flows through a router increases, the amount
of required buffer decreases.
d
Q = W − 2TP · C. (5) This result has practical implications for building routers.
A core router currently has from 10,000 to over 100,000 flows
Because W has a normal distribution, Q has the distribution passing through it at any given time. While the vast ma-
of a normal shifted by a constant (of course, the normal dis- jority of flows are short (e.g. flows with fewer than 100
tribution is restricted to the allowable range for Q). This is packets), the flow length distribution is heavy tailed and
very useful, because we can now pick a buffer size and know the majority of packets at any given time belong to long
immediately the probability that the buffer will underflow flows. As a result, such a router would achieve close to full
and lose throughput. 1
utilization with buffer sizes that are only √10000 = 1% of
Because it is gaussian, we can determine the queue occu-
the delay-bandwidth product. We will verify this result ex-
pancy process if we know its mean and variance. The mean
perimentally in Section 5.2.
is simply the sum of the mean of its constituents. To find
the variance, we’ll assume for convenience that all sawtooths 6
A more precise result could be obtained by using Chernoff
have the same average value (having different values would Bounds instead. We here present the Guassian approxima-
still yield the same results). Each TCP sawtooth can be tion for clarity of presentation.

4. SIZING THE ROUTER BUFFER FOR 50
40 Mbit/s link
80 Mbit/s link
SHORT FLOWS 45
200 Mbit/s link
M/G/1 Model
Not all TCP flows are long-lived; in fact many flows last 40

Average Queue Length E[Q]
only a few packets, never leave slow-start, and so never reach 35
their equilibrium sending rate [4]. Up until now we’ve only
30
considered long-lived TCP flows, and so now we’ll consider
how short TCP flows affect the size of the router buffer. 25

We’re going to find that short flows (TCP and non-TCP) 20
have a much smaller effect than long-lived TCP flows, par-
15
ticularly in a backbone router with a large number of flows.
We will define a short-lived flow to be a TCP flow that 10
never leaves slow-start (e.g. any flow with fewer than 90 5
packets, assuming a typical maximum window size of 65kB).
0
In Section 5.3 we will see that our results hold for short non- 0 10 20 30 40 50 60
TCP flows too (e.g. DNS queries, ICMP, etc.). Length of TCP Flow [pkts]
Consider again the topology in Figure 2 with multiple
senders on separate access links. As has been widely re- Figure 10: The average queue length as a function
ported from measurement, we assume that new short flows of the flow length for ρ = 0.8. The bandwidth has no
arrive according to a Poisson process [16, 17]. In slow-start, impact on the buffer requirement. The upper bound
each flow first sends out two packets, then four, eight, six- given by the M/G/1 model with infinite access link
teen, etc. This is the slow-start algorithm in which the speeds matches the simulation data closely.
sender increases the window-size by one packet for each re-
ceived ACK. If the access links have lower bandwidth than
the bottleneck link, the bursts are spread out and a single packet-by-packet while in an M/G/1 queue the job is only
burst causes no queueing. We assume the worst case where dequeued when the whole job has been processed. If the
access links have infinite speed, bursts arrive intact at the queue is busy, it will overestimate the queue length by half
bottleneck router. the average job size, and so
We will model bursts arriving from many different short
flows at the bottleneck router. Some flows will be sending a ρ E[X 2 ] E[X]
E[Q] = −ρ
burst of two packets, while others might be sending a burst 2(1 − ρ) E[X] 2
of four, eight, or sixteen packets and so on. There will be a
It is interesting to note that the average queue length is
distribution of burst-sizes; and if there is a very large num-
independent of the number of flows and the bandwidth of
ber of flows, we can consider each burst to be independent
the link. It only depends on the load of the link and the
of the other bursts, even of the other bursts in the same
length of the flows.
flow. In this simplified model, the arrival process of bursts
We can validate our model by comparing it with simula-
themselves (as opposed to the arrival of flows) can be as-
tions. Figure 10 shows a plot of the average queue length for
sumed to be Poisson. One might argue that the arrivals are
a fixed load and varying flow lengths, generated using ns2.
not Poisson as a burst is followed by another burst one RTT
Graphs for three different bottleneck link bandwidths (40, 80
later. However under a low load and with many flows, the
and 200 Mb/s) are shown. The model predicts the relation-
buffer will usually empty several times during one RTT and
ship very closely. Perhaps surprisingly, the average queue
is effectively “memoryless” at this time scale.
length peaks when the probability of large bursts is highest,
For instance, let’s assume we have arrivals of flows of a
not necessarily when the average burst size is highest. For
fixed length l. Because of the doubling of the burst lengths
instance, flows of size 14 will generate a larger queue length
in each iteration of slow-start, each flow will arrive in n
than flows of size 16. This is because a flow of 14 packets
bursts of size
generates bursts of Xi = {2, 4, 8} and the largest burst of
Xi = {2, 4, ...2n−1 , R}, size 8 has a probability of 1 . A flow of 16 packets generates
3
bursts of sizes Xi = {2, 4, 8, 4}, where the maximum burst
where R is the remainder, R = l mod (2n − 1). Therefore, 1
length of 8 has a probability of 4 . As the model predicts,
the bursts arrive as a Poisson process, and their lengths
the bandwidth has no effect on queue length, and the mea-
are i.i.d. random variables, equally distributed among
surements for 40, 80 and 200 Mb/s are almost identical. The
{2, 4, ...2n−1 , R}.
gap between model and simulation is due to the fact that the
The router buffer can now be modelled as a simple M/G/1
access links before the bottleneck link space out the packets
queue with a FIFO service discipline. In our case a “job” is
of each burst. Slower access links would produce an even
a burst of packets, and the job size is the number of packets
smaller average queue length.
in a burst. The average number of jobs in an M/G/1 queue
To determine the buffer size we need the probability dis-
is known to be (e.g. [18])
tribution of the queue length, not just its average. This is
ρ more difficult as no closed form result exists for a general
E[N ] = E[X 2 ].
2(1 − ρ) M/G/1 queue length distribution. Instead, we approximate
Here ρ is the load on the link (the ratio of the amount of its tail using the effective bandwidth model [19], which tells
incoming traffic to the link capacity C), and E[X] and E[X 2 ] us that the queue length distribution is
are the first two moments of the burst size. This model will −b
2(1−ρ) E[Xi ]
.
ρ E[X 2 ]
overestimate the queue length because bursts are processed P (Q ≥ b) = e i

This equation is derived in Appendix C. 400
98.0% Utilization
Our goal is to drop very few packets (if a short flow drops 350
99.5% Utilization
99.9% Utilization
a packet, the retransmission significantly increases the flow’s RTTxBW/sqrt(x)
2*RTTxBW/sqrt(x)

Minimum required buffer [pkts]
duration). In other words, we want to choose a buffer size 300
B such that P (Q ≥ B) is small.
A key observation is that - for short flows - the size of 250

the buffer does not depend on the line-rate, the propagation 200
delay of the flows, or the number of flows; it only depends
on the load of the link, and length of the flows. Therefore, a 150
backbone router serving highly aggregated traffic needs the
same amount of buffering to absorb short-lived flows as a 100

router serving only a few clients. Furthermore, because our 50
analysis doesn’t depend on the dynamics of slow-start (only
on the burst-size distribution), it can be easily extended to 0
50 100 150 200 250 300 350 400 450 500
short unresponsive UDP flows.
Number of long-lived flows
In practice, buffers can be made even slower. For our
model and simulation we assumed access links that are faster
than the bottleneck link. There is evidence [4, 20] that Figure 11: Minimum required buffer to achieve 98,
highly aggregated traffic from slow access links in some cases 99.5 and 99.9 percent utilization for an OC3 (155
can lead to bursts being smoothed out completely. In this Mb/s) line with about 80ms average RTT measured
case individual packet arrivals are close to Poisson, result- with ns2 for long-lived TCP flows.
ing in even smaller buffers. The buffer size can be easily
computed with an M/D/1 model by setting Xi = 1.
In summary, short-lived flows require only small buffers. simulations to cases where flows experience only one con-
When there is a mix of short- and long-lived flows, we will see gested link. Network operators usually run backbone links
from simulations and experiments in the next section, that at loads of 10%-30% and as a result packet drops are rare
the short-lived flows contribute very little to the buffering in the Internet backbone. If a single point of congestion is
requirements, and so the buffer size will usually be deter- rare, then it is unlikely that a flow will encounter two or
mined by the number of long-lived flows7 . more congestion points.
We assume that the router maintains a single FIFO queue,
and drops packets from the tail only when the queue is full
5. SIMULATION AND EXPERIMENTAL (i.e. the router does not use RED). Drop-tail is the most
RESULTS widely deployed scheme today. We expect the results to
Up until now we’ve described only theoretical models of extend to RED, because our results rely on the desynchro-
long- and short-lived flows. We now present results to val- nization of TCP flows — something that is more likely with
idate our models. We use two validation methods: simu- RED than with drop-tail. We used TCP Reno with a max-
lation (using ns2), and a network of real Internet routers. imum advertised window size of at least 10000 bytes, and a
The simulations give us the most flexibility: they allow us to 1500 or 1000 byte MTU. The average propagation delay of
explore a range of topologies, link speeds, numbers of flows a TCP flow varied from 25ms to 300ms.
and traffic mixes. On the other hand, the experimental net-
work allows us to compare our results against a network 5.1.1 Simulation Results for Long-lived TCP Flows
of real Internet routers and TCP sources (rather than the Figure 11 simulates an OC3 (155Mb/s) line carrying long-
idealized ones in ns2). It is less flexible and has a limited lived TCP flows. The graph shows the minimum required
number of routers and topologies. Our results are limited to buffer for a given utilization of the line, and compares it with
the finite number of different simulations and experiments the buffer size predicted by the model. For example, our
we can run, and we can’t prove that the results extend to model predicts that for 98% utilization a buffer of RT√n T ×C
any router in the Internet [21]. And so in Section 5.3 we should be sufficient. When the number of long-lived flows
examine the scope and limitations of our results, and what is small the flows are partially synchronized, and the result
further validation steps are needed. doesn’t hold. However – and as can be seen in the graph
Our goal is to persuade a network operator to test our – when the number of flows exceeds 250, the model holds
results by reducing the size of their router buffers by ap- well. We found that in order to attain 99.9% utilization, we
proximately 99%, and checking that the utilization and drop needed buffers twice as big; just as the model predicts.
rates don’t change noticeably. Until that time, we have to We found similar results to hold over a wide range of
rely on a more limited set of experiments. settings whenever there are a large number of flows, there
is little or no synchronization, and the average congestion
5.1 NS2 Simulations window is above two. If the average congestion window
We ran over 15,000 ns2 simulations, each one simulating is smaller than two, flows encounter frequent timeouts and
several minutes of network traffic through a router to verify more buffering is required [22].
our model over a range of possible settings. We limit our In our simulations and experiments we looked at three
7
For a distribution of flows we define short flows and long other commonly used performance metrics, to see their
flows as flows that are in slow-start and congestion avoidance effect on buffer size:
mode respectively. This means that flows may transition
from short to long during their existence.

250
40 Mbit/s link 2000 Minimum buffer for 95% utilization
80 Mbit/s link 1 x RTT*BW/sqrt(n)
200 Mbit/s link 0.5 x RTT*BW/sqrt(n)
M/G/1 Model p=0.01
200

Minimum required buffer [pkts]
Minimum Required Buffer

1500

150

1000
100

500
50

0 0
0 10 20 30 40 50 60 0 50 100 150 200 250 300 350 400 450 500
Length of TCP Flow [pkts] Number of long-lived flows

Figure 13: Buffer requirements for traffic mix with
Figure 12: The minimum required buffer that in-
different flow lengths, measured from a ns2 simula-
creases the Average Flow Completion Time (AFCT)
tion.
by not more than 12.5% vs infinite buffers for short
flow traffic.
load ρ 1, the AFCT is minimized when we have infinite
• Packet loss If we reduce buffer size, we can expect buffers, because there will be no packet drops and therefore
packet loss to increase. The loss rate of a TCP flow no retransmissions.
is a function of the flow’s window size and can be ap- We take as a benchmark the AFCT with infinite buffers,
proximated to l = 0.76 (see [22]). From Equation 1, we
w2
then find the increase in AFCT as a function of buffer size.
know the sum of the window sizes is RT T × C + B. If For example, Figure 12 shows the minimum required buffer
B is made very small, then the window size halves, in- so that the AFCT is increased by no more than 12.5%. Ex-
creasing loss by a factor of four. This is not necessarily perimental data is from ns2 experiments for 40, 80 and 200
a problem. TCP uses loss as a useful signal to indi- Mb/s and a load of 0.8. Our model, with P (Q > B) = 0.025,
cate congestion; and TCP’s loss rate is very low (one is plotted in the graph. The bound predicted by the M/G/1
packet per multiple round-trip times). More impor- model closely matches the simulation results.
tantly, as we show below, flows complete sooner with The key result here is that the amount of buffering needed
smaller buffers than with large buffers. One might ar- does not depend on the number of flows, the bandwidth or
gue that other applications that do not use TCP are the round-trip time. It only depends on the load of the link
adversely affected by loss (e.g. online gaming or me- and the length of the bursts. For the same traffic mix of only
dia streaming), however these applications are typi- short flows, a future generation 1 Tb/s core router needs the
cally even more sensitive to queueing delay. same amount of buffering as a local 10 Mb/s router today.
• Goodput While 100% utilization is achievable, good-
put is always below 100% because of retransmissions.
5.1.3 Mixes of Short- and Long-Lived Flows
With increased loss, goodput is reduced, but by a very In practice, routers transport a mix of short and long
small amount, as long as we have buffers equal or flows; the exact distribution of flow lengths varies from net-
greater than RT√n .
T ×C work to network, and over time. This makes it impossible
to measure every possible scenario, and so we give a gen-
• Fairness Small buffers reduce fairness among flows. eral idea of how the flow mix influences the buffer size. The
First, a smaller buffer means all flows have a smaller good news √ that long flows dominate, and a buffer size of
is
round-trip time, and their sending rate is higher. With RT T × C/ n will suffice when we have a large number of
large buffers, all round-trip times increase and so the flows. Better still, we’ll see that the AFCT for the short
relative difference of rates will decrease. While overflows is lower than if we used the usual rule-of-thumb.
buffering would increase fairness, it also increases flow In our experiments the short flows always slow-down the
completion times for all flows. A second effect is that long flows because of their more aggressive multiplicative
timeouts are more likely with small buffers. We did increase, causing the long flows to reduce their window-size.
not investigate how timeouts affect fairness in detail, Figures 13 and 14 show a mix of flows over a 400 Mb/s link.
however in our ns2 simulations it seemed to be only The total bandwidth of all arriving short flows is about 80
minimally affected by buffer size. Mb/s or 20% of the total capacity. The number of long flows
was varied from 1 to 500. During the time of the experiment,
5.1.2 Short Flows these long flows attempted to take up all the bandwidth left
We will use the commonly used metric for short flows: available by short flows. In practice, they never consumed
the flow completion time, defined as the time from when more than 80% of the bandwidth as the rest would be taken
the first packet is sent until the last packet reaches the des- by the more aggressive short flows.
tination. In particular, we will measure the average flow As we expected, with a small number of flows, the flows
completion time (AFCT). We are interested in the tradeoff are partially synchronized. With more than 200 long-lived
between buffer size and AFCT. In general, for a link with a flows, the synchronization has largely disappeared. The

700 due to its additive increase behavior above a certain window
Average completion time for a 14 pkt TCP flow
size. Traffic transported by high-speed routers on commer-
600 cial networks today [4, 24] has 10’s of 1000’s of concurrent
flows and we believe this is unlikely to change in the future.
500 An uncongested router (i.e. ρ 1) can be modeled using
the short-flow model presented in section 4 which often leads
400
to even lower buffer requirements. Such small buffers may
penalize very long flows as they will be forced into congestion
300
avoidance early even though bandwidth is still available. If
200
we want to allow a single flow to take up to 1/n of the√
bandwidth, we always need buffers of (RT T × C)/ n, even
100 at a low link utilization.
AFCT of a 14 packet flow (RTT*BW Buffers) We found that our general result holds for different flow
AFCT of a 14 packet flow (RTT*BW/sqrt(n) Buffers)
0 length distributions if at least 10% of the traffic is from long
0 50 100 150 200 250 300 350 400 450 500
flows. Otherwise, short flow effects sometimes dominate.
Number of long-lived flows
Measurements on commercial networks [4] suggest that over
Figure 14: Average flow completion times with a 90% of the traffic is from long flows. It seems safe to as-
√
buffer size of (RT T × C)/ n, compared with a buffer sume that long flows drive buffer requirements in backbone
size RT T × C. routers.

5.2 Measurements on a Physical Router
graph shows that the long flows dominate the flow size. While simulation captures most characteristics or router-
If we want√ 95% utilization, then we need a buffer close to TCP interaction, we verified our model by running experi-
RT T × C/ n. 8 This means we can ignore the short flows ments on a real backbone router with traffic generated by
when sizing the buffer. Of course, this doesn’t tell us how the real TCP sources.
short-lived flows are faring — they might be shutout by the The router was a Cisco GSR 12410 [25] with a 4 x OC3
long flows, and have increased AFCT. But Figure 14 shows POS “Engine 0” line card that switches IP packets using POS
that this is not the case. In this ns2 simulation, the average
√ (PPP over Sonet) at 155Mb/s. The router has both input
flow completion time is much shorter with RT T × C/ n and output queues, although no input queueing took place
buffers than with RT T × C sized buffers. This is because in our experiments, as the total throughput of the router
the queueing delay is lower. So by reducing the buffer size, was far below the maximum capacity of the switching fabric.
we can still achieve the 100% utilization and decrease the TCP traffic was generated using the Harpoon traffic genera-
completion times for shorter flows. tor [26] on Linux and BSD machines, and aggregated using
a second Cisco GSR 12410 router with Gigabit Ethernet line
5.1.4 Pareto Distributed Flow Lengths cards. Utilization measurements were done using SNMP on
Real network traffic contains a broad range of flow lengths. the receiving end, and compared to Netflow records [27].
The flow length distribution is known to be heavy tailed
[4] and in our experiments we used Pareto distributions to TCP Router Buffer Link Utilization (%)
RT T ×BW
model it. As before, we define short flows to be those still Flows √
n
Pkts RAM Model Sim. Exp.
in slow start. 100 0.5 x 64 1 Mbit 96.9% 94.7% 94.9%
For Pareto distributed flows on a congested router (i.e. 100 1x 129 2 Mbit 99.9% 99.3% 98.1%
ρ ≈ 1), the model holds and we can achieve close to 100% 100 2x 258 4 Mbit 100% 99.9% 99.8%
throughput with buffers of a small multiple of (RT T ×
√ 100 3x 387 8 Mbit 100% 99.8% 99.7%
C)/ n.9 For example in an ns2 simulation of a 155 Mb/s 200 0.5 x 46 1 Mbit 98.8% 97.0% 98.6%
¯
line, RT T ≈ 100ms) we measured 100-200 simultaneous 200 1x 91 2 Mbit 99.9% 99.2% 99.7%
flows and achieved a utilization of 99% with a buffer of only 200 2x 182 4 Mbit 100% 99.8% 99.8%
165 packets. 200 3x 273 4 Mbit 100% 100% 99.8%
It has been pointed out [23] that in a network with low la- 300 0.5 x 37 512 kb 99.5% 98.6% 99.6%
tency, fast access links and no limit on the TCP window size, 300 1x 74 1 Mbit 100% 99.3% 99.8%
there would be very few concurrent flows. In such a network, 300 2x 148 2 Mbit 100% 99.9% 99.8%
a single very heavy flow could hog all the bandwidth for a 300 3x 222 4 Mbit 100% 100% 100%
short period of time and then terminate. But this is un- 400 0.5 x 32 512 kb 99.7% 99.2% 99.5%
likely in practice, unless an operator allows a single user to 400 1x 64 1 Mbit 100% 99.8% 100%
saturate their network. And so long as backbone networks 400 2x 128 2 Mbit 100% 100% 100%
are orders of magnitude faster than access networks, few 400 3x 192 4 Mbit 100% 100% 99.9%
users will be able to saturate the backbone anyway. Even
if they could, TCP is not capable of utilizing a link quickly Figure 15: Comparison of our model, ns2 simulation
8 and experimental results for buffer requirements of
Here n is the number of active long flows at any given time,
not the total number of flows. a Cisco GSR 12410 OC3 linecard.
9
The number of long flows n for sizing the buffer was found
by measuring the number of flows in congestion avoidance
mode at each instant and visually selecting a robust mini- 5.2.1 Long Flows
mum. Figure 15 shows the results of measurements from the

GSR 12410 router. The router memory was adjusted by is no reverse path congestion, which would likely have an
limiting the length of the interface queue on the outgoing effect on TCP-buffer interactions [28]. With these assump-
interface. The buffer size is given as a multiple of RT√n ,
T ×C tions, our simplified network topology is fairly general. In
the number of packets and the size of the RAM device that an arbitrary network, flows may pass through other routers
would be needed. We subtracted the size of the internal before and after the bottleneck link. However, as we assume
FIFO on the line-card (see Section 5.2.2). Model is the lower- only a single point of congestion, no packet loss and little
bound on the utilization predicted by the model. Sim. and traffic shaping will occur on previous links in the network.
Exp. are the utilization as measured by a simulation with We focus on TCP as it is the main traffic type on the
ns2 and on the physical router respectively. For 100 and 200 internet today. Constant rate UDP sources (e.g. online
flows there is, as we expect, some synchronization. Above games) or single packet sources with Poisson arrivals (e.g.
that the model predicts the utilization correctly within the DNS) can be modelled using our short flow model and the
measurement accuracy of about ±0.1%. ns2 sometimes pre- results for mixes of flows still hold. But to understand traffic
dicts a lower utilization than we found in practice. We at- composed mostly of non-TCP packets would require further
tribute this to more synchronization between flows in the study.
simulations than in the real network. Our model assumes there is no upper bound on the con-
The key result here is that model, simulation and experi- gestion window. In reality, TCP implementations have max-
ment all agree that a router buffer should have a size equal imum window sizes as low as 6 packets [29]. Window sizes
to approximately RT√n , as opposed to RT T × C (which
T ×C above 64kByte require use of a scaling option [30] which
is rarely used. Our results still hold as flows with limited
in this case would be 1291 packets). window sizes require even smaller router buffers [1].
We did run some simulations using Random Early De-
1
Exp. Cisco GSR if-queue tection [12] and this had an effect on flow synchronization
Exp. Cisco GSR buffers
Model M/G/1 PS
for a small number of flows. Aggregates of a large number
Model M/G/1 FIFO (> 500) of flows with varying RTTs are not synchronized
and RED tends to have little or no effect on buffer require-
0.1 ments. However early drop can slightly increase the required
buffer since it uses buffers less efficiently.
P(Q > x)

FIFO
There was no visible impact of varying the latency other
than its direct effect of varying the bandwidth-delay prod-
uct.
0.01
Congestion can also be caused by denial of service (DOS)
[31] attacks that attempt to flood hosts or routers with large
amounts of network traffic. Understanding how to make
routers robust against DOS attacks is beyond the scope of
0.001 this paper, however we did not find any direct benefit of
0 50 100 150 200 250 300 350 400
larger buffers for resistance to DOS attacks.
Queue length [pkts]

Figure 16: Experimental, Simulation and Model 6. RELATED WORK
prediction of a router’s queue occupancy for a Cisco Villamizar and Song report the RT T × BW rule in [1],
GSR 12410 router. in which the authors measure link utilization of a 40 Mb/s
network with 1, 4 and 8 long-lived TCP flows for different
buffer sizes. They find that for FIFO dropping discipline and
5.2.2 Short Flows very large maximum advertised TCP congestion windows it
In Section 4 we used an M/G/1 model to predict the buffer is necessary to have buffers of RT T ×C to guarantee full link
size we would need for short-lived, bursty TCP flows. To utilization. We reproduced their results using ns2 and can
verify our model, we generated lots of short-lived flows and confirm them for the same setup. With such a small number
measured the probability distribution of the queue length of of flows, and large congestion windows, the flows are almost
the GSR 12410 router. Figure 16 shows the results and the fully synchronized and have the same buffer requirement as
comparison with the model, which match remarkably well.10 a single flow.
Morris [32] investigates buffer requirements for up to 1500
5.3 Scope of our Results and Future Work long-lived flows over a link of 10 Mb/s with 25ms latency.
The results we present in this paper assume only a single He concludes that the minimum amount of buffering needed
point of congestion on a flow’s path. We don’t believe our is a small multiple of the number of flows, and points out
results would change much if a percentage of the flows ex- that for a bandwidth-delay product of 217 packets, each flow
perienced congestion on multiple links, however we have not has only a fraction of a packet in transit at any time. Many
investigated this. A single point of congestion means there flows are in timeout, which adversely effects utilization and
fairness. We repeated the experiment in ns2 and obtained
10
The results match very closely if we assume the router similar results. However for a typical router used by a car-
under-reports the queue length by 43 packets. We learned rier or ISP, this has limited implications. Users with fast
from the manufacturer that the line-card has an undocu- access links will need several packets outstanding to achieve
mented 128kByte transmit FIFO. In our setup, 64 kBytes
are used to queue packets in an internal FIFO which, with adequate performance. Users with very slow access links
an MTU of 1500 bytes, accounts exactly for the 43 packet (e.g. 32kb/s modem users or 9.6kb/s GSM mobile access)
difference. need additional buffers in the network so they have sufficient

packets outstanding. However this additional buffer should for setting up and running the measurements on a physical
be at the ends of the access link, e.g. the modem bank at the router in their WAIL testbed; and Sally Floyd and Frank
local ISP, or GSM gateway of a cellular carrier. We believe Kelly for useful disucssions. Matthew Holliman’s feedback
that overbuffering the core router that serves different users on long flows led to the central limit theorem argument.
would be the wrong approach, as overbuffering increases la-
tency for everyone and is also difficult to implement at high
line-rates. Instead the access devices that serve slow, last-
9. REFERENCES
mile access links of under 1Mb/s should continue to include [1] C. Villamizar and C. Song. High performance tcp in
a few packets worth of buffering for each link. ansnet. ACM Computer Communications Review,
With line speeds increasing and the MTU size staying 24(5):45–60, 1994 1994.
constant, we would also assume this issue to become less [2] Cisco line cards.
relevant in the future. http://www.cisco.com/en/US/products/hw/modules/
Avrachenkov et al [33] present a fixed point model for ps2710/products data sheets list.html.
utilization (for long flows) and flow completion times (for [3] R. Bush and D. Meyer. RFC 3439: Some internet
short flows). They model short flows using an M/M/1/K architectural guidelines and philosophy, December
model that only accounts for flows but not for bursts. In 2003.
their long flow model they use an analytical model of TCP [4] C. J. Fraleigh. Provisioning Internet Backbone
that is affected by the buffer through the RTT. As the model Networks to Support Latency Sensitive Applications.
requires fixed point iteration to calculate values for specific PhD thesis, Stanford University, Department of
settings and only one simulation result is given, we can not Electrical Engineering, June 2002.
directly compare their results with ours. [5] D. Ferguson. [e2e] Queue size of routers. Posting to
Garetto and Towsley [34] describe a model for queue the end-to-end mailing list, January 21, 2003.
lengths in routers with a load below one that is similar to [6] S. H. Low, F. Paganini, J. Wang, S. Adlakha, and
our model in section 4. The key difference is that the au- J. C. Doyle. Dynamics of tcp/red and a scalable
thors model bursts as batch arrivals in an M [k] /M/1 model control. In Proceedings of IEEE INFOCOM 2002, New
(as opposed to our model that models bursts by varying York, USA, June 2002.
the job length in a M/G/1 model). It accommodates both
[7] S. Iyer, R. R. Kompella, and N. McKeown. Analysis of
slow-start and congestion avoidance mode, however it lacks
a memory architecture for fast packet buffers. In
a closed form solution. In the end the authors obtain queue
Proceedings of IEEE High Performance Switching and
distributions that are very similar to ours.
Routing, Dallas, Texas, May 2001.
[8] C. Dovrolis. [e2e] Queue size of routers. Posting to the
7. CONCLUSION end-to-end mailing list, January 17, 2003.
We believe that the buffers in backbone routers are much [9] S. Shenker, L. Zhang, and D. Clark. Some
larger than they need to be — possibly by two orders of observations on the dynamics of a congestion control
magnitude. If our results are right, they have consequences algorithm. ACM Computer Communications Review,
for the design of backbone routers. While we have evidence pages 30–39, Oct 1990.
that buffers can be made smaller, we haven’t tested the hy- [10] The network simulator - ns-2.
pothesis in a real operational network. It is a little difficult http://www.isi.edu/nsnam/ns/.
to persuade the operator of a functioning, profitable network [11] Y. J. Anna Gilbert and N. McKeown. Congestion
to take the risk and remove 99% of their buffers. But that control and periodic behavior. In LANMAN
has to be the next step, and we see the results presented in Workshop, March 2001.
this paper as a first step towards persuading an operator to [12] S. Floyd and V. Jacobson. Random early detection
try it. gateways for congestion avoidance. IEEE/ACM
If an operator verifies our results, or at least demonstrates Transactions on Networking, 1(4):397–413, 1993.
that much smaller buffers work fine, it still remains to per- [13] L. Zhang and D. D. Clark. Oscillating behaviour of
suade the manufacturers of routers to build routers with network traffic: A case study simulation.
fewer buffers. In the short-term, this is difficult too. In Internetworking: Research and Experience, 1:101–112,
a competitive market-place, it is not obvious that a router 1990.
vendor would feel comfortable building a router with 1% of
[14] L. Qiu, Y. Zhang, and S. Keshav. Understanding the
the buffers of its competitors. For historical reasons, the net-
performance of many tcp flows. Comput. Networks,
work operator is likely to buy the router with larger buffers,
37(3-4):277–306, 2001.
even if they are unnecessary.
[15] G. Iannaccone, M. May, and C. Diot. Aggregate traffic
Eventually, if routers continue to be built using the current
performance with active queue management and drop
rule-of-thumb, it will become very difficult to build linecards
from tail. SIGCOMM Comput. Commun. Rev.,
from commercial memory chips. And so in the end, necessity
31(3):4–13, 2001.
may force buffers to be smaller. At least, if our results are
true, we know the routers will continue to work just fine, [16] V. Paxson and S. Floyd. Wide area traffic: the failure
and the network utilization is unlikely to be affected. of Poisson modeling. IEEE/ACM Transactions on
Networking, 3(3):226–244, 1995.
[17] A. Feldmann, A. C. Gilbert, and W. Willinger. Data
8. ACKNOWLEDGMENTS networks as cascades: Investigating the multifractal
The authors would like to thank Joel Sommers and Profes- nature of internet WAN traffic. In SIGCOMM, pages
sor Paul Barford from the University of Wisconsin-Madison 42–55, 1998.

[18] R. W. Wolff. Stochastic Modelling and the Theory of APPENDIX
Queues, chapter 8. Prentice Hall, October 1989.
[19] F. P. Kelly. Notes on Effective Bandwidth, pages A. SUMMARY OF TCP BEHAVIOR
141–168. Oxford University Press, 1996. Let’s briefly review the basic TCP rules necessary to un-
[20] J. Cao, W. Cleveland, D. Lin, and D. Sun. Internet derstand this paper. We will present a very simplified de-
traffic tends to poisson and independent as the load scription of TCP Reno. For a more detailed and more com-
increases. Technical report, Bell Labs, 2001. prehensive presentation, please refer to e.g. [35].
[21] S. Floyd and V. Paxson. Difficulties in simulating the The sending behavior of the TCP sender is determined by
internet. IEEE/ACM Transactions on Networking, its state (either slow-start or congestion avoidance) and the
February 2001. congestion window, W . Initially, the sender is in slow-start
[22] R. Morris. Scalable tcp congestion control. In and W is set to two. Packets are sent until there are W
Proceedings of IEEE INFOCOM 2000, Tel Aviv, USA, packets outstanding. Upon receiving a packet, the receiver
March 2000. sends back an acknowledgement (ACK). The sender is not
allowed to have more than W outstanding packets for which
[23] S. B. Fredj, T. Bonald, A. Prouti`re, G. R´gni´, and
e e e
it has not yet received ACKs.
J. Roberts. Statistical bandwidth sharing: a study of
While in slow-start, the sender increases W for each ac-
congestion at flow level. In Proceedings of SIGCOMM
knowledgement it receives. As packets take one RTT to
2001, San Diego, USA, August 2001.
travel to the receiver and back this means W effectively dou-
[24] Personal communication with stanford networking on bles every RTT. If the sender detects a lost packet, it halves
characteristics of residential traffic. its window and enters congestion avoidance. In congestion
[25] Cisco 12000 series routers. avoidance, it only increases W by 1/W for each acknowledg-
http://www.cisco.com/en/US/products/hw/ ment it receives. This effectively increases W by one every
routers/ps167/. RTT. Future losses always halve the window but the flow
[26] J. Sommers, H. Kim, and P. Barford. Harpoon: A stays in CA mode.
flow-level traffic generator for router and network test.
In Proceedings of ACM SIGMETRICS, 2004. (to No loss: Wnew = 2Wold
Slow-Start:
appear). Loss: Wnew = W2 , enter CA
old

[27] I. Cisco Systems. Netflow services solution guido, July No loss: Wnew = Wold + 1
CA:
2001. http://www.cisco.com/. Loss: Wnew = W2
old

[28] L. Zhang, S. Shenker, and D. D. Clark. Observations In both states, if several packets are not acknowledged in
on the dynamics of a congestion control algorithm: time, the sender can also trigger a timeout. It then goes
The effects of two-way traffic. In Proceedings of ACM back to the slow-start mode and the initial window size.
SIGCOMM, pages 133–147, September 1991. Note that while in congestion avoidance, the window size
[29] Microsoft. Tcp/ip and nbt configuration parameters typically exhibits a sawtooth pattern. The window size in-
for windows xp. Microsoft Knowledge Base Article - creases linearly until the first loss. It then sharply halves the
314053, November 4, 2003. window size, and pauses to receive more ACKs (because the
[30] K. McCloghrie and M. T. Rose. RFC 1213: window size has halved, the number of allowed outstanding
Management information base for network packets is halved, and so the sender must wait for them to
management of TCP/IP-based internets:MIB-II, be acknowledged before continuing). The sender then starts
March 1991. Status: STANDARD. increasing the window size again.
[31] A. Hussain, J. Heidemann, and C. Papadopoulos. A The sending rate of the TCP flow is the window size di-
framework for classifying denial of service attacks. In vided by the round trip time:
Proceedings of ACM SIGCOMM, August 2003.
W W
[32] R. Morris. Tcp behavior with many flows. In R= = (7)
Proceedings of the IEEE International Conference on RT T 2Tp + TQ
Network Protocols, Atlanta, Georgia, October 1997. where Tp is the one-way, end-to-end propagation delay of
[33] K. Avrachenkov, U. Ayesta, E. Altman, P. Nain, and a packet and TQ is the queueing delay it encounters at the
C. Barakat. The effect of router buffer size on the tcp router (we assume a single point of congestion and no queue-
performance. In Proceedings of the LONIIS Workshop ing delay for ACKs).
on Telecommunication Networks and Teletraffic
Theory, pages 116–121, St.Petersburg, Russia, Januar
2002.
B. BEHAVIOR OF A SINGLE LONG TCP
[34] M. Garetto and D. Towsley. Modeling, simulation and FLOW
measurements of queueing delay under long-tail Consider a single long TCP flow, going through a single
internet traffic. In Proceedings of SIGMETRICS 2003, router of buffer size equal to the delay-bandwidth product
San Diego, USA, June 2003. (Figure 2). It is often assumed that the round-trip time
[35] W. R. Stevens. TCP Illustrated, Volume 1 - The of such a flow is nearly constant. Using this assumption,
Protocols. Addison Wesley, 1994. since the window size is incremented at each round-trip time
when there is no loss, the increase in window size should be
linear with time. In other words, the sawtooth pattern of
the window size should be triangular.
However, the round-trip time is not constant, and there-
fore the sawtooth pattern is not exactly linear, as seen in

Figure 4. This becomes especially noticeable as link capaci- C. QUEUE DISTRIBUTION USING
ties, and therefore delay-bandwidth products, become larger EFFECTIVE BANDWIDTH
and larger. Note that if the sawtooth was linear and the
round-trip time was constant, the queue size increase would The goal of this appendix is to model the behavior of
be parabolic by integration. On the contrary, we will see the router queue when there are several senders with short
below that it is concave and therefore sub-linear. TCP flows. As explained in Section 4, each of these short
Let’s introduce a fluid model for this non-linear sawtooth TCP flows sends a given number of bursts. We assume that
behavior. Let Q(t) denote the packet queue size at time t, all these bursts are independent, and by considering all the
with Q(0) = 0. As defined before, 2Tp is the round-trip independent bursts of all the flows, we model the arrival
propagation time, and C is the router output link capacity. process to the router as the cumulative arrival of these in-
dependent bursts. Let A(t) denote the cumulative amount
For this fluid model, assume that the propagation time is
negligible compared to the time it takes to fill up the buffer. of traffic arrived to the router at time t, N (t) the number
Then, in the regularly increasing part of the sawtooth, the of bursts arrived before t, and Xi the size of each of these
bursts. Then:
window size increases by 1 packet every RTT (Appendix A),
and therefore N (t)
X
1 1 A(t) = Xi .
˙
W (t) = = . i=1
RT T 2Tp + Q(t)/C
We assume that the bursts arrive as a Poisson process N (t)
In addition the window size is linked to the queue size by
of rate ν, and that their burst size follows a distribution func-
W (t) = 2Tp C + Q(t). tion F . The queue is serviced with a capacity C. Therefore,
we use an M/G/1 model for the job size in the router queue.
Joining both equations yields Of course, the traffic arrival rate λ is the product of the
C burst arrival rate by the average burst size:
˙
W (t) = . (8)
W (t) λ = νE[X].
Hence
The effective bandwidth theory describes the characteris-
W 2 (t)
“ ”
d 2
tics of a traffic source, and in many cases can be a powerful
=C tool to derive properties that are otherwise hard to compute.
dt For more information we refer the reader to [19], from which
most of the results below are adapted. Consider a cumula-
W 2 (t) = 2Ct + W (0). tive traffic arrival function A(t), where A(t) has stationary
When the sawtooth pattern starts, Q(0) = 0, thus increments. Its effective bandwidth is defined as:
1 h i
W (0) = 2Tp C + Q(0) = 2Tp C. α(s, t) = logE esA(t) .
st
We finally get a simple model for the increase of the window
In our case, A(t) has i.i.d. increments. It is a special case
size in the sawtooth,
of L´vy process, and its effective bandwidth has a simpler
e
form (section 2.4 of [19]):
p
W (t) = 2Ct + (2Tp C)2 . (9)
Z
Therefore, as t Tp , the queue size behaves in a square- 1 λ h i
α(s) = (esx − 1)νdF (x) = E esX − 1 . (11)
root fashion, instead of a linear one, as commonly believed. s sE[X]
The queue size behaves in a similar fashion,
We can now use Cram´r’s estimate to model the tail of the
e
distribution of the queue length Q (section 3.2 of [19]). This
p
Q(t) = W (t) − 2Tp C = 2Ct + (2Tp C)2 − 2Tp C. (10)
estimate assumes an infinite buffer size. For Cram´r’s esti-
e
As an application, it is now possible to determine the pe- mate there has to exist a constant κ such that the effective
riodicity T of the sawtooth pattern. The maximum buffer bandwidth is the link capacity:
size B is reached when Q(T ) = B, i.e.
α(κ) = C (12)
B2
T = + 2Tp B. Also the derivative α (s) has to exist at κ. Both are satisfied
2C
in our case as λ < C and α(s) is differentiable for any s > 0.
When the buffer size is equal to the bandwidth-delay prod-
Cramer’s estimate is given by
uct (B = 2Tp C), we get
C − α(0) −κb
B2 P (Q ≥ b) ≈ e (13)
T = 1.5 . κα (κ)
C
For instance, in Figure 4, the modelled sawtooth period is To calculate κ we need to simplify Equation (12). We ap-
2
T = 1.5 142 = 30.2, which fits the observed value. proximate the exponential function by its Taylor Series.
1000
s2 2 s3 3
esX = 1 + sX + X + X + O(X 4 ) (14)
2 6
We present solutions for the 2nd and 3rd order approxi-
mation. In our experience the 2nd order approximation is
sufficient to estimate the required buffer.

Second Order Approximation We substitute Equa-
tion (14) into Equation (11) and obtain for the 2nd order
approximation:
s2
» –
λ
α(s) = E 1 + sX + X 2 − 1
sE[X] 2
λs E[X 2 ]
= λ+
2 E[X]
λ E[X 2 ]
α (s) =
2 E[X]
The load ρ is deﬁned as the ratio of the arrival rate λ by the
capacity C:
λ
ρ= .
C
Using Equation (12), we get:
2(1 − ρ) E[X]
κ=
ρ E[X 2 ]
Finally, for the queue size distribution we obtain
λ
ρ
−λ
P (Q ≥ b) ≈ 2(1−ρ) E[X] λ E[X 2 ]
e−bκ = e−bκ
ρ E[X 2 ] 2 E[X]

Third Order Approximation. We substitute Equa-
tion (14) into Equation (11) and obtain for the 3rd order
approximation:
s2 s3
» –
λ
α(s) = E 1 + sX + X 2 + X 3 − 1
sE[X] 2 6
λs E[X 2 ] λs2 E[X 3 ]
= λ+ +
2 E[X] 6 E[X]
λ E[X 2 ] λs E[X 3 ]
α (s) = +
2 E[X] 3 E[X]
λ
Again we now need to solve α(κ) = C where ρ = C
. We
obtain the quadratic equation.
3E[X 2 ] 6(1 − ρ) E[X]
κ2 + κ − =0
E[X 3 ] ρ E[X 3 ]

and for κ
s„ «2
3 E[X 2 ] 3 E[X 2 ] 6(1 − ρ) E[X]
κ=− + +
2 E[X 3 ] 2 E[X 3 ] ρ E[X 3 ]
For the queue size distribution we ﬁnd.
λ
ρ
−λ
P (Q ≥ b) ≈ e−bκ
λκ E[X 2 ] λκ2 E[X 3 ]
2 E[X]
+ 3 E[X]

1−ρ E[X]
= κ2
e−bκ
ρ κ
E[X 2 ] + E[X 3 ]
2 3

4) sizing router buffers

More Related Content

What's hot (20)

Similar to 4) sizing router buffers (20)

4) sizing router buffers