How NOT to Measure Latency, Gil Tene, London, Oct. 2013

©2013 Azul Systems, Inc.

How NOT to Measure
Latency
An attempt to confer wisdom...
Gil Tene, CTO & co-Founder, Azul Systems


High level agenda


High level agenda
Some latency behavior background


High level agenda
The pitfalls of using “statistics”


High level agenda
Latency “philosophy” questions


High level agenda
The Coordinated Omission Problem


High level agenda
Some useful tools


High level agenda
Some useful tools
Use tools for bragging


About me: Gil Tene


About me: Gil Tene
co-founder, CTO @Azul
Systems


About me: Gil Tene
Systems
Have been working on “think
different” GC approaches
since 2002


About me: Gil Tene
Systems
since 2002
* working on real-world trash compaction issues, circa 2004


About me: Gil Tene
Systems
since 2002
Created Pauseless & C4 core
GC algorithms (Tene, Wolf)


About me: Gil Tene
Systems
since 2002
A Long history building
Virtual & Physical Machines,
Operating Systems,
Enterprise apps, etc...


About me: Gil Tene
Systems
since 2002
A Long history building
Virtual & Physical Machines,
Operating Systems,
Enterprise apps, etc...
JCP EC Member...


About Azul


About Azul
We make scalable Virtual
Machines


About Azul
Machines
Have built “whatever it takes to
get job done” since 2002


About Azul
Machines
3 generations of custom SMP
Multi-core HW (Vega)


About Azul
Machines
Vega


About Azul
Machines
Zing: Pure software for
commodity x86
Vega


About Azul
Machines
commodity x86
Vega
C4


About Azul
Machines
commodity x86
Known for Low Latency,
Consistent execution, and Large
data set excellence
Vega
C4


Common fallacies


Common fallacies
Computers run application code continuously


Common fallacies
Response time can be measured as work units/time


Common fallacies
Response time exhibits a normal distribution


Common fallacies
“Glitches” or “Semi-random omissions” in
measurement don’t have a big effect.


Common fallacies
“Glitches” or “Semi-random omissions” in
measurement don’t have a big effect.
Wrong!
Wrong!
Wrong!
Wrong!


A classic look at response
time behavior
Response time as a function of load
source: IBM CICS server documentation, “understanding response times”


A classic look at response
time behavior
Response time as a function of load
source: IBM CICS server documentation, “understanding response times”
Average?
Max?
Median?
90%?
99.9%


Response time over time
When we measure behavior over time, we often see:
source: ZOHO QEngine White Paper: performance testing report analysis
Interpreting Results
From this graph it is possible to identify peaks in the response time over a period of time. Helps to
identify issues when server is continuously loaded for long duration of time.
Response Time Graph (per Transaction)
Response Time versus Elapsed Time report indicates the average response time of the transactions
over the elapsed time of the load test as shown.


Response time over time
When we measure behavior over time, we often see:
source: ZOHO QEngine White Paper: performance testing report analysis
Interpreting Results
From this graph it is possible to identify peaks in the response time over a period of time. Helps to
identify issues when server is continuously loaded for long duration of time.
Response Time Graph (per Transaction)
Response Time versus Elapsed Time report indicates the average response time of the transactions
over the elapsed time of the load test as shown.
“Hiccups”


0"
1000"
2000"
3000"
4000"
5000"
6000"
7000"
8000"
9000"
0" 20" 40" 60" 80" 100" 120" 140" 160" 180"
Hiccup&Dura*on&(msec)&
&Elapsed&Time&(sec)&
Hiccups&by&Time&Interval&
Max"per"Interval" 99%" 99.90%" 99.99%" Max"
What happened here?


0"
1000"
2000"
3000"
4000"
5000"
6000"
7000"
8000"
9000"
0" 20" 40" 60" 80" 100" 120" 140" 160" 180"
What happened here?
“Hiccups”


0"
1000"
2000"
3000"
4000"
5000"
6000"
7000"
8000"
9000"
0" 20" 40" 60" 80" 100" 120" 140" 160" 180"
What happened here?
Source: Gil running an idle program and suspending it ﬁve times in the middle
“Hiccups”


0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
25"
Hiccups&by&Percen*le&Distribu*on&
The real world (a low latency example)


0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
25"
99%‘ile is ~60 usec


0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
25"
99%‘ile is ~60 usec
Max is ~30,000%
higher than “typical”


Hiccups are [typically]
strongly multi-modal


They don’t look anything like a normal distribution


They usually look like periodic freezes


A complete shift from one mode/behavior to another


Mode A: “good”.


Mode A: “good”.
Mode B: “Somewhat bad”


Mode A: “good”.
Mode C: “terrible”, ...


Mode A: “good”.
Mode C: “terrible”, ...
....


Common ways people deal with hiccups


Averages and Standard Deviation


Averages and Standard Deviation
Always
Wrong!


Better ways people can deal with hiccups
Actually measuring percentiles0"
20"
40"
60"
80"
100"
0" 500" 1000" 1500" 2000" 2500"
Hiccup&Dura*on&(ms
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Hiccups"by"Percen?le" SLA"


20"
40"
60"
80"
100"
0" 500" 1000" 1500" 2000" 2500"
Hiccup&Dura*on&(ms
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Response Time
Percentile
plot line


20"
40"
60"
80"
100"
0" 500" 1000" 1500" 2000" 2500"
Hiccup&Dura*on&(ms
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Requirements
Response Time
Percentile
plot line


Requirements
Why we measure latency and response times
to begin with...


Latency tells us how long
something took


something took
But what do we WANT the latency to be?


something took
What do we want the latency to BEHAVE like?


something took
Latency requirements are usually a PASS/FAIL test of
some predeﬁned criteria


something took
Different applications have different needs


something took
Requirements should reﬂect application needs


something took
Requirements should reﬂect application needs
Measurements should provide data to evaluate
requirements


The Olympics
aka “ring the bell ﬁrst”


The Olympics
Goal: Get gold medals


The Olympics
Need to be faster than everyone else at SOME races


The Olympics
Ok to be slower in some, as long as fastest at some
(the average speed doesn’t matter)


The Olympics
Ok to not even ﬁnish or compete (the worst case and
99%‘ile don’t matter)


The Olympics
Ok to not even ﬁnish or compete (the worst case and
99%‘ile don’t matter)
Different strategies can apply. E.g. compete in only 3
races to not risk burning out, or compete in 8 races
in hope of winning two


Pacemakers
aka “hard” real time


Pacemakers
Goal: Keep heart beating


Pacemakers
Need to never be slower than X


Pacemakers
“Your heart will keep beating 99.9% of the time” is
not reassuring


Pacemakers
not reassuring
Having a good average and a nice standard deviation
don’t matter or help


Pacemakers
not reassuring
Having a good average and a nice standard deviation
don’t matter or help
The worst case is all that matters


“Low Latency” Trading
aka “soft” real time


Goal A: Be fast enough to make some good plays


Goal B: Contain risk and exposure while making plays


E.g. want to “typically” react within 200 usec.


But can’t afford to hold open position for 20 msec, or
react to 30 msec stale information


So we want a very good “typical” (median, 50%‘ile)


So we want a very good “typical” (median, 50%‘ile)
But we also need a reasonable Max, or 99.99%‘ile


Interactive applications
aka “squishy” real time


Goal: Keep users happy enough to not complain/leave


Need to have “typically snappy” behavior


Ok to have occasional longer times, but not too high, and
not too often


not too often
Example: 90% of responses should be below 0.2 sec, 99%
should be below 0.5 sec, 99.9 should be better than 2
seconds. And a >10 second response should never happen.


not too often
Example: 90% of responses should be below 0.2 sec, 99%
should be below 0.5 sec, 99.9 should be better than 2
seconds. And a >10 second response should never happen.
Remember: A single user may have 100s of interactions
per session...


Establishing Requirements
an interactive interview (or thought) process


Q: What are your latency requirements?


A: We need an avg. response of 20 msec


Q: Ok. Typical/average of 20 msec... So what is the worst case requirement?


A: We don’t have one


Q: So it’s ok for some things to take more than 5 hours?


A: No way in H%%&!


A: No way in H%%&!
Q: So I’ll write down “5 hours worst case...”


A: No way in H%%&!
A: No. That’s not what I said. Make that “nothing worse than 100 msec”


A: No way in H%%&!
Q: Are you sure? Even if it’s only two times a day?


A: No way in H%%&!
Q: Are you sure? Even if it’s only two times a day?
A: Ok... Make it “nothing worse than 2 seconds...”


Ok. So we need a typical of 20msec, and a worst case of 2 seconds. How often is it ok
to have a 1 second response?


A: (Annoyed) I thought you said only a few times a day


Q: That was for the worst case. But if half the results are better than 20 msec, is it ok
for the other half to be just short of 2 seconds? What % of the time are you willing to
take a 1 second, or a half second hiccup? Or some other level?


A: Oh. Let’s see. We have to better than 50 msec 90% of the time, or we’ll be losing
money even when we are fast the rest of the time. We need to be better than 500
msec 99.9% of the time, or our customers will complain and go elsewhere


Now we have a service level expectation:


50% better than 20 msec


99.9% better than 500 msec


99.9% better than 500 msec
100% better than 2 seconds


Latency does not live in a vacuum


Remember this?
How much load can this system handle?


Remember this?
What the
marketing
benchmarks
will say


Remember this?
What the
marketing
benchmarks
will say
Where users
complain


Remember this?
Where the
sysadmin
is willing
to go
What the
marketing
benchmarks
will say
Where users
complain


Remember this?
Where the
sysadmin
is willing
to go
What the
marketing
benchmarks
will say
Where users
complain
Sustainable
Throughput
Level


Sustainable Throughput: The
throughput achieved while safely
maintaining service levels


Sustainable Throughput: The
throughput achieved while safely
maintaining service levels
Unsustainable
Throughout


Comparing behavior under different throughputs
and/or conﬁgurations
0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
0#
20#
40#
60#
80#
100#
120#
!Dura&on!(msec)!
!
!
Percen&le!
Dura&on!by!Percen&le!Distribu&on!
Setup#D#
Setup#E#
Setup#F#
Setup#A#
Setup#B#
Setup#C#


0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
0#
10#
20#
30#
40#
50#
60#
70#
80#
90#
Latency((milliseconds)(
(
(
Percen3le(
Latency(by(Percen3le(Distribu3on(
Comparing latency behavior under different throughputs,
conﬁgurations
HotSpot&10K&&&
HotSpot&15K&
HotSpot&5K&
HotSpot&1K&
Zing&15K&
Zing&10K&
Zing&5K&
Zing&1K&
&
Required&
Service&Level&


Instance capacity test: “Fat Portal”
HotSpot CMS: Peaks at ~3GB / 45 concurrent users
* LifeRay portal on JBoss @ 99.9% SLA of 5 second response times


Instance capacity test: “Fat Portal”
C4: still smooth @ 800 concurrent users
* LifeRay portal on JBoss @ 99.9% SLA of 5 second response times


The coordinated omission problem
An accidental conspiracy...


Common Example A (load testing):
build/buy load tester to measure system behavior
each “client” issues requests one by one at a certain rate
measure and log response time for each request
results log used to produce histograms, percentiles, etc.


Common Example A (load testing):
build/buy load tester to measure system behavior
each “client” issues requests one by one at a certain rate
measure and log response time for each request
results log used to produce histograms, percentiles, etc.
So what’s wrong with that?
works well only when all responses ﬁt within rate interval
technique includes implicit “automatic backoff” and coordination
But requirements interested in random, uncoordinated requests


Common Example B (monitoring):
System monitors and records each transaction latency
Latency measured between start and end of each operation
keeps observed latency stats of some sort (log, histogram, etc.)


Common Example B (monitoring):
System monitors and records each transaction latency
Latency measured between start and end of each operation
keeps observed latency stats of some sort (log, histogram, etc.)
So what’s wrong with that?
works correctly well only when no queuing occurs
Long operations only get measured once
delays outside of timing window do not get measured at all
queued operations are measured wrong


Common Example B:
Coordinated Omission in Monitoring Code
Long operations only get measured once
delays outside of timing window do not get measured at all

How bad can this get?
Elapsed Time
System easily handles
100 requests/sec
Responds to each
in 1msec

System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec

System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
How would you characterize this system?

Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec

Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
Avg. is 50 sec.
over next 100 sec

Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
Avg. is 50 sec.
over next 100 sec
Overall Average response time is ~25 sec.

Avg. is 1 msec
over 1st 100 sec
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
~50%‘ile is 1 msec ~75%‘ile is 50 sec 99.99%‘ile is ~100sec
Avg. is 50 sec.
over next 100 sec
Overall Average response time is ~25 sec.

Measurement in practice
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec

System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
Naïve Characterization

System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 @ 1msec 1 @ 100 second

System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 @ 1msec 1 @ 100 second
99.99%‘ile is 1 msec! Average. is 10.9msec! Std. Dev. is 0.99sec!
(should be ~100sec) (should be ~25 sec)

Proper measurement
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each

Proper measurement
System Stalled
for 100 Sec
Elapsed Time
100 requests/sec
Responds to each
in 1msec
10,000 results
Varying linearly
from 100 sec
to 10 msec
10,000 results
@ 1 msec each
Coordinated
Omission


It is MUCH more common than you may
think...


JMeter makes this mistake... (so do others)
Before
Correction
After
Correcting
for
Omission


The “real” world


Results were
collected by a
single client
thread


26.182 seconds
represents 1.29%
of the total time
Results were
collected by a
single client
thread


99%‘ile MUST be at least 0.29%
of total time (1.29% - 1%)
which would be 5.9 seconds
26.182 seconds
represents 1.29%
of the total time
Results were
collected by a
single client
thread


99%‘ile MUST be at least 0.29%
of total time (1.29% - 1%)
which would be 5.9 seconds
26.182 seconds
represents 1.29%
of the total time
wrong by a factor
of 1,000x
Results were
collected by a
single client
thread


A world record SPECjEnterprise2010 result


305.197 seconds
represents 8.4% of
the timing run


The max is 762 (!!!)
standard deviations
away from the mean
305.197 seconds
represents 8.4% of
the timing run


0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
Max=7995.39*
0#
1000#
2000#
3000#
4000#
5000#
6000#
7000#
8000#
9000#
Latency*(msec)*
*
*
Percen7le*
Uncorrected*Latency*by*Percen7le*Distribu7on*
0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
Max=7995.39*
0#
1000#
2000#
3000#
4000#
5000#
6000#
7000#
8000#
9000#
Latency*(msec)*
*
*
Percen7le*
Corrected*Latency*by*Percen7le*Distribu7on*
Real World Coordinated Omission effects
Before
Correction
After
Correction


0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
Max=7995.39*
0#
1000#
2000#
3000#
4000#
5000#
6000#
7000#
8000#
9000#
Latency*(msec)*
*
*
Percen7le*
Uncorrected*Latency*by*Percen7le*Distribu7on*
0%# 90%# 99%# 99.9%# 99.99%# 99.999%#
Max=7995.39*
0#
1000#
2000#
3000#
4000#
5000#
6000#
7000#
8000#
9000#
Latency*(msec)*
*
*
Percen7le*
Corrected*Latency*by*Percen7le*Distribu7on*
Before
Correction
After
Correction
Wrong
by 7x


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
Uncorrected
Data


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Uncorrected
Data


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Uncorrected
Data
Corrected for
Coordinated
Omission


(Why I care)
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Zing#Base#
Zing#Corrected1#
Zing#Corrected2#


(Why I care)
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Zing#Base#
Zing#Corrected1#
Zing#Corrected2#
Zing


(Why I care)
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Zing#Base#
Zing#Corrected1#
Zing#Corrected2#
Zing
“other”
JVM


(Why I care)
0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
0#
100000#
200000#
300000#
400000#
500000#
600000#
!Dura&on!(usec)!
!
!
Percen&le!
HotSpot#Base#
HotSpot#Corrected1#
HotSpot#Corrected2#
Zing#Base#
Zing#Corrected1#
Zing#Corrected2#
A ~2500x
difference in
reported
percentile levels
for the problem
that Zing
eliminates
Zing
“other”
JVM


Suggestions


Suggestions
Whatever your measurement technique is, TEST IT.


Suggestions
Run your measurement method against an artiﬁcial
system that creates hypothetical pauses scenarios. See if
your reported results agree with how you would describe
that system behavior


Suggestions
Don’t waste time analyzing until you establish sanity


Suggestions
Don’t EVER use or derive from std. deviation


Suggestions
ALWAYS measure Max time. Consider what it means...
Be suspicious.


Suggestions
ALWAYS measure Max time. Consider what it means...
Be suspicious.
Measure %‘iles. Lots of them.


Some Tools


HdrHistogram


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
Max=137.472+
0#
20#
40#
60#
80#
100#
120#
140#
160#
Latency+(msec)+
+
+
Percen8le+
Latency+by+Percen8le+Distribu8on+
HdrHistogram
If you want to be able to produce graphs like this...
You need both good dynamic range
and good resolution


HdrHistogram background


Goal: Collect data for good latency characterization...
Including acceptable precision at and between varying percentile levels


Existing alternatives
Record all data, analyze later (e.g. sort and get 99.9%‘ile).
Record in traditional histograms


Existing alternatives
Record all data, analyze later (e.g. sort and get 99.9%‘ile).
Record in traditional histograms
Traditional Histograms: Linear bins, Logarithmic bins, or
Arbitrary bins
Linear requires lots of storage to cover range with good resolution
Logarithmic covers wide range but has terrible precisions
Arbitrary is.... arbitrary. Works only when you have a good feel for the
interesting parts of the value range


HdrHistogram
A High Dynamic Range Histogram
Covers a configurable dynamic value range
At configurable precision (expressed as number of significant digits)


HdrHistogram
For Example:
Track values between 1 microsecond and 1 hour
With 3 decimal points of resolution


HdrHistogram
For Example:
Built-in [optional] compensation for Coordinated
Omission


HdrHistogram
For Example:
Built-in [optional] compensation for Coordinated
Omission
Open Source
On github, released to the public domain, creative commons CC0


HdrHistogram
Fixed cost in both space and time
Built with “latency sensitive” applications in mind
Recording values does not allocate or grow any data structures
Recording values uses a ﬁxed computation to determine location (no
searches, no variability in recording cost, FAST)
Even iterating through histogram can be done with no allocation


HdrHistogram
Fixed cost in both space and time
Built with “latency sensitive” applications in mind
Recording values does not allocate or grow any data structures
Recording values uses a ﬁxed computation to determine location (no
searches, no variability in recording cost, FAST)
Even iterating through histogram can be done with no allocation
Internals work like a “ﬂoating point” data structure
“Exponent” and “Mantissa”
Exponent determines “Mantissa bucket” to use
“Mantissa buckets” provide linear value range for a given exponent.
Each have enough linear entries to support required precision


HdrHistogram


HdrHistogram
Provides tools for iteration
Linear, Logarithmic, Percentile


HdrHistogram
Supports percentile iterators
Practical due to high dynamic range


HdrHistogram
Supports percentile iterators
Practical due to high dynamic range
Convenient percentile output
10% intervals between 0 and 50%
5% intervals between 50% and 75%
2.5% intervals between 75% and 87.5%...
Very useful for feeding percentile
distribution graphs...


0%# 90%# 99%# 99.9%# 99.99%# 99.999%# 99.9999%#
Max=137.472+
0#
20#
40#
60#
80#
100#
120#
140#
160#
Latency+(msec)+
+
+
Percen8le+
Latency+by+Percen8le+Distribu8on+
HdrHistogram


jHiccup


Incontinuities in Java platform execution
0"
200"
400"
600"
800"
1000"
1200"
1400"
1600"
1800"
0" 200" 400" 600" 800" 1000" 1200" 1400" 1600" 1800"
Hiccups"by"Time"Interval"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=1665.024&
0"
200"
400"
600"
800"
1000"
1200"
1400"
1600"
1800"
&
&
Percen*le&
Hiccups"by"Percen@le"Distribu@on"


jHiccup
A tool for capturing and displaying platform hiccups
Records any observed non-continuity of the underlying platform
Plots results in simple, consistent format


jHiccup
Simple, non-intrusive
As simple as adding jHiccup.jar as a java agent:
% java -javaagent=jHiccup.jar myApp myﬂags
or attaching jHiccup to a running process:
% jHiccup -p <pid>
Adds a background thread that samples time @ 1000/sec into an
HdrHistogram


jHiccup
Simple, non-intrusive
As simple as adding jHiccup.jar as a java agent:
% java -javaagent=jHiccup.jar myApp myﬂags
or attaching jHiccup to a running process:
% jHiccup -p <pid>
Adds a background thread that samples time @ 1000/sec into an
HdrHistogram
Open Source. Released to the public domain


Telco App Example
0"
20"
40"
60"
80"
100"
120"
140"
0" 500" 1000" 1500" 2000" 2500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&


Telco App Example
0"
20"
40"
60"
80"
100"
120"
140"
0" 500" 1000" 1500" 2000" 2500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Max Time per
interval


Telco App Example
0"
20"
40"
60"
80"
100"
120"
140"
0" 500" 1000" 1500" 2000" 2500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Max Time per
interval
Hiccup
duration at
percentile
levels


Telco App Example
0"
20"
40"
60"
80"
100"
120"
140"
0" 500" 1000" 1500" 2000" 2500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
0"
20"
40"
60"
80"
100"
120"
140"
&
&
Percen*le&
Optional SLA
plotting
Max Time per
interval
Hiccup
duration at
percentile
levels


Fun with jHiccup


Oracle HotSpot CMS, 1GB in an 8GB heap
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
0" 500" 1000" 1500" 2000" 2500" 3000" 3500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=13156.352&
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
&
&
Percen*le&
Zing 5, 1GB in an 8GB heap
0"
5"
10"
15"
20"
25"
0" 500" 1000" 1500" 2000" 2500" 3000" 3500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"
Max=20.384&
0"
5"
10"
15"
20"
25"
&
&
Percen*le&


Oracle HotSpot CMS, 1GB in an 8GB heap
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
0" 500" 1000" 1500" 2000" 2500" 3000" 3500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=13156.352&
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
&
&
Percen*le&
Zing 5, 1GB in an 8GB heap
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
0" 500" 1000" 1500" 2000" 2500" 3000" 3500"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"Max=20.384&
0"
2000"
4000"
6000"
8000"
10000"
12000"
14000"
&
&
Percen*le&
Drawn to scale


Good for both
“squishy” real time
(human response times)
and
“soft” real time
(low latency software systems)


0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=22.656&
0"
5"
10"
15"
20"
25"
&
&
Percen*le&
Oracle HotSpot (pure newgen)
Low latency trading application


0"
0.2"
0.4"
0.6"
0.8"
1"
1.2"
1.4"
1.6"
1.8"
0" 100" 200" 300" 400" 500" 600"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=1.568&
0"
0.2"
0.4"
0.6"
0.8"
1"
1.2"
1.4"
1.6"
1.8"
&
&
Percen*le&
0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=22.656&
0"
5"
10"
15"
20"
25"
&
&
Percen*le&
Oracle HotSpot (pure newgen) Zing
Low latency trading application


Low latency - Drawn to scale
Oracle HotSpot (pure newgen) Zing
0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=1.568&
0"
5"
10"
15"
20"
25"
&
&
Percen*le&
0"
5"
10"
15"
20"
25"
0" 100" 200" 300" 400" 500" 600"
0%" 90%" 99%" 99.9%" 99.99%" 99.999%"
Max=22.656&
0"
5"
10"
15"
20"
25"
&
&
Percen*le&


Shameless bragging


Zing


Zing
A JVM for Linux/x86 servers


Zing
ELIMINATES Garbage Collection as a concern for
enterprise applications


Zing
Very wide operating range: Used in both low latency
and large scale enterprise application spaces


Zing
Very wide operating range: Used in both low latency
and large scale enterprise application spaces
Decouples scale metrics from response time concerns
Transaction rate, data set size, concurrent users,
heap size, allocation rate, mutation rate, etc.


What is Zing good for?


If you have a server-based Java application


And you are running on Linux


And you use using more than ~300MB of memory


And you use using more than ~300MB of memory
Then Zing will likely deliver superior behavior
metrics


Where Zing shines


Where Zing shines
Low latency
Eliminate behavior blips down to the sub-millisecond-units level


Where Zing shines
Low latency
Machine-to-machine “stuff”
Support higher *sustainable* throughput (the one that meets
SLAs)


Where Zing shines
Low latency
SLAs)
Human response times
Eliminate user-annoying response time blips. Multi-second and
even fraction-of-a-second blips will be completely gone.
Support larger memory JVMs *if needed* (e.g. larger virtual user
counts, or larger cache, in-memory state, or consolidating multiple
instances)


Where Zing shines
Low latency
SLAs)
Human response times
Eliminate user-annoying response time blips. Multi-second and
even fraction-of-a-second blips will be completely gone.
Support larger memory JVMs *if needed* (e.g. larger virtual user
counts, or larger cache, in-memory state, or consolidating multiple
instances)
“Large” data and in-memory analytics


Takeaways


Takeaways
Standard Deviation and application latency should
never show up on the same page...


Takeaways
If you haven’t stated percentiles and a Max, you
haven’t speciﬁed your requirements


Takeaways
Measuring throughput without latency behavior is
[usually] meaningless


Takeaways
Measuring throughput without latency behavior is
[usually] meaningless
Mistakes in measurement/analysis can cause orders-
of-magnitude errors and lead to bad business
decisions


Q & A
http://www.azulsystems.com
http://www.jhiccup.com
http://giltene.github.com/HdrHistogram

How NOT to Measure Latency, Gil Tene, London, Oct. 2013

More Related Content

What's hot (20)

Similar to How NOT to Measure Latency, Gil Tene, London, Oct. 2013 (20)

More from Azul Systems Inc. (20)

Recently uploaded (20)

How NOT to Measure Latency, Gil Tene, London, Oct. 2013