How NOT to Measure Latency

©2015 Azul Systems, Inc.
How NOT to
Measure Latency
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations/
latency-response-time

Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

The “Oh S@%#!” talk
Gil Tene, CTO & co-Founder, Azul Systems
@giltene

About me: Gil Tene
co-founder, CTO @Azul
Systems

Have been working on
“think different” GC
approaches since 2002

A Long history building
Virtual & Physical
Machines, Operating
Systems, Enterprise apps,
etc...

I also depress people by
pulling the wool up from
over their eyes…
* working on real-world trash compaction issues, circa 2004

Latency Behavior
Latency: The time it took one operation to happen

Each operation occurrence has its own latency

What we care about is how latency behaves

Behavior is a lot more than “the common case was X”

We like to look at pretty charts…
95%’lie
The “We only want to show good things” chart

A real world, real time example

A real world, real time example
So this is a better picture. Right?

Why do we tend to avoid plotting Max latency?
Because no other %’ile will be visible on the same chart…

I like to rant about latency…

#LatencyTipOfTheDay:
If you are not measuring and/or
plotting Max, what are you hiding
(from)?

What (TF) does the Average
of the 95%’lie mean?

What (TF) does the Average
of the 95%’lie mean?
Lets do the same with 100%’ile; Suppose we a set of
100%’ile values for each minute:

[1, 0, 3, 1, 601, 4, 2, 8, 0, 3, 3, 1, 1, 0, 2]

“The average 100%’ile over the past 15 minutes was 42”

Same nonsense applies to any other %’lie

You can't average percentiles.
Period.

Percentiles Matter

Is the 99%’lie “rare”?

99%’lie: a good indicator, right?
What are the chances of a single web page
view experiencing >99%’lie latency of:
- A single search engine node?
- A single Key/Value store node?
- A single Database node?
- A single CDN request?

MOST page loads will experience
the 99%'lie server response

Which HTTP response time metric is more
“representative” of user experience?
The 95%’lie or the 99.9%’lie

Gauging user experience
Example: If a typical user session involves 5 page
loads, averaging 40 resources per page.
- How many of our users will NOT experience
something worse than the 95%’lie of http requests?
Answer: ~0.003%
- How may of our users will experience at least one
response that is longer than the 99.9%’lie?
Answer: ~18%

Gauging user experience
Example: If a typical user session involves 5 page
loads, averaging 40 resources per page.
- What http response percentile will be experienced
by the 95%’ile of users?
Answer: ~99.97%
- What http response percentile will be experienced
by the 99%’ile of users
Answer: ~99.995%

Median Server Response Time:
The number that 99.9999999999%
of page views can be worse than

Why don’t we have response
time or latency stats with
multiple 9s in them???

You can’t average
percentiles…
And you also can’t get an
hour’s 99.999%’lie out of lots
of 10 second interval 99%’lie
reports…

You can’t average percentiles…
And you also can’t get an hour’s
99.999%’lie out of lots
of 10 second interval 99%’lie reports…
Check out HdrHistogram
It lets you have nice things….
0"
20"
40"
60"
80"
100"
120"
0" 500" 1000" 1500" 2000" 2500"
Hiccup&Dura*on&(mse
&Elapsed&Time&(sec)&
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"Hiccup&Dura*on&(msec)&
&
&
Percen*le&
Hiccups&by&Percen*le&Distribu*on&
Hiccups"by"Percen?le" SLA"

Latency “wishful thinking”
We know how to compute
averages & std. deviation, etc.

Wouldn’t it be nice if latency
had a normal distribution?

The average, 90%’lie, 99%’lie,
std. deviation, etc. can give us
a “feel” for the rest of the
distribution, right?

If 99% of the stuff behaves
well, how bad can the rest be,
really?

The real world: latency distribution
0%! 90%! 99%!
0!
0.05!
0.1!
0.15!
0.2!
0.25!
0.3!
0.35!
0.4!
0.45!
0.5!
Latency(msec)!
!
!
Percentile!
Latency by Percentile Distribution!

0%! 90%! 99%! 99.9%!
0!
0.5!
1!
1.5!
2!
2.5!
3!
3.5!
4!
4.5!
5!
Latency(msec)!
!
!
Percentile!

0%! 90%! 99%! 99.9%! 99.99%! 99.999%! 99.9999%!
0!
10!
20!
30!
40!
50!
60!
Latency(msec)!
!
!
Percentile!

Dispelling standard deviation
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
10#
20#
30#
40#
50#
Latency(((msec)(
(
(
Percen/le(
Latency(by(Percen/le(Distribu/on(
A#
B#
C#
D#
E#
F#

Dispelling standard deviation
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
10#
20#
30#
40#
50#
Latency(((msec)(
(
(
Percen/le(
Latency(by(Percen/le(Distribu/on(
A#
B#
C#
D#
E#
F#
Mean = 0.06 msec

Std. Deviation (𝞂) = 0.21msec
99.999% = 38.66msec
In a normal distribution,
These are NOT normal distributions
~184 σ (!!!) away from the mean
the 99.999%’ile falls within 4.5 σ

The coordinated omission problem
An accidental conspiracy...
The lie in the 99%’lies

The coordinated omission
problem
Common Example A (load testing):

each “client” issues requests at a certain rate

measure/log response time for each request

So what’s wrong with that?

works only if ALL responses ﬁt within interval

implicit “automatic back off” coordination

Common Example B:

Coordinated Omission in Monitoring Code
Long operations only get measured once
delays outside of timing window do not get measured at all

How bad can this get?
40
Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
Avg. is 50 sec.
over next 100 sec
Overall Average response time is ~25 sec.

Measurement in prac8ce
41
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
What actually gets measured?
(should be ~50sec) (should be ~100 sec)
50%‘ile is 1 msec 75%‘lie is 1 msec 99.99%‘lie is 1 msec
10,000 measurements
@ 1 msec each
1 measurement
@ 100 sec
Overall Average is 10.9 msec (!!!)

Proper measurement
42
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each

Proper measurement
43
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
Coordinated
Omission
1 msec 1 msec
1 result
@ 100 sec

“Be?er” can look “Worse”
44
System Slowed
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 @ 1msec 10,000 @ 5 msec
50%‘ile is 1 msec 75%‘lie is 2.5msec 99.99%‘lie is ~5msec
Still easily handles
100 requests/sec
Responds to each
in 5 msec
(stalled shows 1 msec) (stalled shows 1 msec)

“Correc8on”: “Chea8ng Twice”
45
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
Coordinated
Omission
1 msec 1 msec
9,999 additional results
@ 1 msec each
1 result
@ 100 sec
“correction”

Response Time vs. Service Time

Service Time vs. Response Time

Coordinated Omission
Usually
makes something that you think is a
Response Time metric
only represent
the Service Time component

Response Time vs. Service Time @2K/sec

How “real” people react

Service Time, 90K/s vs 80K/s

Response Time, 90K/s vs 80K/s

Response Time, 90K/s vs 80K/s (to scale)

Latency doesn’t live in a vacuum

Sustainable Throughput:
The throughput achieved while
safely maintaining service levels

Comparing behavior under different throughputs
and/or conﬁgurations
0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
0#
20#
40#
60#
80#
100#
120#
!Dura&on!(msec)!
!
!
Percen&le!
Dura&on!by!Percen&le!Distribu&on!
Setup#D#
Setup#E#
Setup#F#
Setup#A#
Setup#B#
Setup#C#

Comparing response time or
latency behaviors

System A @90K/s & 85K/s vs.
System B @90K/s & 85K/s
Wrong Place to Look:
They both “suck” at >85K/sec

System A 85K/s vs. System B 85K/s
Looks good, but still
the wrong place to look

System A @40K/s vs. System B @40K/s
More interesting…
What can we do with this?

E.g. if “99%’ile < 5msec” was a goal:
System B delivers similar 99%’ile and superior
99.9%’ile+ while carrying 4x the throughput

E.g. if “99.9%’ile < 10msec” was a goal:
System B delivers similar 99%’ile and 99.9%’ile
while carrying 10x the throughput

System A @2k thru 80k

System A @2k thru 70k

System B @20k thru 70k

System A & System B @2k thru 70k

System A & System B
10K/s thru 60K/s
System A @ 10K, 20K, 40K, 60K
System B @20K, 40K, 60K
Lots of conclusions can be drawn from the above…
E.g. System B delivers a consistent 100x reduction in the
rate of occurrence of >20msec response times

System A: 200-1400 msec stalls
System B drawn to scale
Service timeResponse Time Response TimeService time

This is Your Load on System A
This is Your Load on System B
Any Questions?
A simple visual summary

http://www.azulsystems.com
Any Questions?

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/latency-
response-time

How NOT to Measure Latency

More Related Content

What's hot (20)

Similar to How NOT to Measure Latency (20)

More from C4Media (20)

Recently uploaded (20)

How NOT to Measure Latency