Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013

Nick Galbreath http://client9.com/20130501 @ngalbreath
Care and Feeding of
Large Scale
Graphite
InstallationsNick Galbreath ★ IPONWEB
DevOpsDays ★ Austin Texas ★ 2013-04-30

http://client9.com/20130501

Who is nickg?

(that's online advertising infrastructure)

• Over One Billion Points collected daily.
• In "many" independent Graphite
clusters.

Who cares?
• Making it easy to create, analyze and
share data can change your
organization
• Making a data-driven culture
• Empowering developers, operations,
qa, security and business to be more
conﬁdent in the changes they make.

What is it you say
you do here?
• Your job is likely invisible to rest of the
organization
• invisible things aren't valued
• so make what you do visible

Why Graphite?
• Many innovations in each part of the
stack
• But, it's the Full Stack that really makes
it special.
• On-disk layout to UI to API to... the
community around it.

Sharing in Caring
• Allows data to be easily accessed
• And easily shared. This makes it
diﬀerent than many monitoring
solutions.
• It's your own in-house mashup
generator.

The Big Picture
• It's a database
• But not ACID
• And without all the database tools

Installation
• 3 python-twistd servers
• "carbon-cache"
• "cache-aggregator"
• "carbon-relay"
• Apache / Django Web UI and API
• Uses SQLite3/MySQL for
dashboards / events

Be Current
• Don't use the OS default version.
• Newer point releases of graphite have
signiﬁcant improvements in storage
engine and webui/api
• It's 100% Python so
"building it yourself" shouldn't to hard.
• pip install works and is current.

The Documentation
• everyone complains about it
• historically bad, but getting a lot better
• Switched locations, but not all
searchengines are updated to use:
http://graphite.readthedocs.org/
• Source code is quite good, so RTFS

Storage Engine

What is it?
• Storage engine
• Handles reads, writes and creates of a
single metric to a fixed size file.
• One file, kinda dumb (good).
• Here's the API:
https://github.com/graphite-project/
whisper/blob/master/whisper.py

Graphite Math
• About 12 bytes per point.
• Store 1 minute points for 1 month and
15 minutes for 11 months.
• (60×24×30 + 4×24×30×11) ×12 =
878kB
• If you can keep all your points in
memory, then magic!

Disk Layout
• Each metric create a directory tree
server123.myapp.logins.failed
• Makes 3 directories
• This creates a very branchy directory
structure
• This has good and bad points.

Middleware
carbon-cache
Part 1

Carbon Cache
• Metrics go in
• csv ﬁle or python pickle format
• TCP
• Metrics go to disk
• whisper

Write Buffer
• Most important feature is write
buffering to protect the disk.
• Data is buffered and written out once
per minute (or so).
• But

The Cache
• It's a write cache.
• Once data is written,
it's out of the cache
• In other words, the cache is
metrics not on disk.
• If the cache dies, you lose metrics
• (btw: the read cache is the os disk
cache)

New Metrics
• New metrics are created automatically
• But, it is very expensive.
• MAX_CREATES_PER_MINUTE.=.50
• Saves your disk, but new metrics will
"pile up" in cache.
• May take 10m+ for your metrics to start
ﬂowing....

FALLOCATE
• WHISPER_FALLOCATE_CREATE.=.True
• Linux&Kernel&>=&2.6.23
• fallocate is used to preallocate blocks
to a file. For filesystems which
support the fallocate system call, this
is done quickly by allocating blocks
and marking them as uninitialized,
requiring no IO to the data blocks.
This is much faster than creating a
file by filling it with zeros.
• https://bugs.launchpad.net/whisper/
+bug/957827

Limit the Size
Limit the size of the cache to avoid
swapping or becoming CPU bound.Sorts
and serving cache queries gets more
expensive as the cache grows.
Use the value "inf" (infinity) for an
unlimited cache size.
MAX_CACHE_SIZE = inf
No!.Infinity.does.not.exist.on.your.system!
.&

Graphite for Graphite
By&default,&carbon&itself&will&log&
statistics&(such&as&a&count,
metricsReceived)&with&the&top&level&prefix&
of&'carbon'&at&an&interval&of&60
seconds.&Set&CARBON_METRIC_INTERVAL&to&0&to&
disable&instrumentation
CARBON_METRIC_PREFIX&=&carbon
CARBON_METRIC_INTERVAL&=&60

Stats on
Stats!

Middleware
carbon-aggregator
Part 2

Pre-Aggregation
• Sum or Average metrics based on
wildcards and regexps
• Helps eliminate very slow queries on
webui
• You can emit the ﬁnal sum & all the
individual components or just the ﬁnal
sum (via blacklists)

destination
• r/w to localhost
• split metrics to other aggregators
• Design your own system

Along the way
• renaming of metrics
• whitelist and blacklist of aggregation
and metrics

Also...
• has support for broadcasting data to
multiple downstream caches
• but.. never used it.. and seems at odds
with the next middleware

Middleware
carbon-relay
Part 3

It's a Router!
• Consistent Hashing (Sharding)
• Or more rule-based routing
• Output to multiple carbon servers
have not really used it much, but should
work similarly to scale outs of
memcache, redis

Middleware
StatsD
Part 4

StatsD
• https://github.com/etsy/statsd/
• nodejs based but lots of other
implementations
• Receives UDP, send graphite-
compatible output, ﬂushed periodically.
• Aggregation for all by default
• Beside sum, also can compute other
basic statistics (mean, 90% percentile),
do sampling, have counters, etc.

StatsD use case
• It's UDP based, so it excels at
embedding a client inside the
application
• UDP can't block or break the sending
application
• Not so good for bulk metrics
• Use both! Can work together with
aggregator.

Of Note
• https://github.com/armon/statsite
• Need to look at this more
• c + libev based
• modern time series algorithms
• very ﬂexible output

Backups
Do you really need them?
http://bit.ly/11sPhNz

Backup
• Doing naive backup causes graphite
performance goes to crap.
• File system cache is trashed
• Metrics are not written to disk (lag)
• If OOM occurs then you lose metrics.

Do you need to save
everything?

nice is
good
http://www.beenthereyet.net/nice-france
If you are doing your own backup....

ionice is better
IONICE(1) User Commands IONICE(1)
NAME
ionice - set or get process I/O scheduling class and priority
SYNOPSIS
ionice [-c class] [-n level] [-t] -p PID...
ionice [-c class] [-n level] [-t] command [argument...]
DESCRIPTION
This program sets or gets the I/O scheduling class and priority for a
program. If no arguments or just -p is given, ionice will query the current I/O
scheduling class and priority for that process.
When command is given, ionice will run this command with the given
arguments. If no class is specified, then command will be executed with the
"best-effort" scheduling class. The default priority level is 4.
NOTES
Linux supports I/O scheduling priorities and classes since 2.6.13 with the
CFQ I/O scheduler.
util-linux July 2011 IONICE(1)

Even Better
• Just write the metrics to two graphite
servers in your client
• Script to copy / resync "holes" when
restoring.

Monitoring

WebUI
• Hey, it's a web server
• do all the usual stuﬀ
• Ask for known stats,
• check for 200
• check for valid json output

Mistakes in URL or use of
functions cause Server 500

Old Stats
• Don't forget to kill oﬀ old metrics
• no updates in X days? kill.
• Exercise in "ﬁnd" left to reader

MySQL
• If you use SQLite3 -- uhh nothing to
monitor
• If you use MySQL -- use the regular
suspects
• And don't forget to backup!!

CPU is stable
• ... except for apache usage
• consider moving apache to separate
machine

Disk is Sensitive
Competing with other
processes for disk does this

Means less written to disk
Metrics updated

Dangerous build up
in cache
Cache Size

Tune Apache
• By default, your Apache install is likely
to be "unlimited" in CPU and Memory
usage.
• Select a wildcard metric for a long time
period can easily turn a httpd process
in 1GB. (this seems like a bug actually)
• OOM death.

/version/
• Yes, ending "/" is required.
• Ok not that exciting but easy check

/metrics/expand/
• /metrics/expand/?query=server*
• {"results": ["server001", "server002", ... ]}

/events/
• Ad-Hoc Events that don't deserve their
own metric type.
• has tags, time, and text
• Stored in SQLite3 by default by the
webapp.
• Rest UI is primitive

The WebUI
• it's "ok".. good for experiments
• You will want to make your own
dashboard.
• Good news! The API is a URL, so it's
very easy.

WebUI Dashboards
• The WebUI has a dashboard feature for
loading and saving graphs
• It saves data in SQLite3 by default
• Since it's there people will use it
• So hack to remove it or, switch to
MySQL.

Granularity
• Like RRDTool, the resolution of the
graph depends on number of pixels
used. No sub-pixel rendering!
• Rapid spikes can be "averaged away"
in week-long views in small graphs.

Vertical Line
Technology
• Easy to make horizontal lines
• Not so clear how to make
ad-hoc vertical lines

Turn "time since" into
events
drawAsInfinite(
removeAboveValue(
keepLastValue(
YOURMETRIC
),
120
)
)

Turn
Version Numbers
into Events
drawAsInfinite(
removeBelowValue(
derivative(keepLastValue(
YOURMETRIC))
,0.1)
)

Arbitrary Lines
drawAsInfinite(
removeBelowValue(
removeAboveValue(
time("time"),
timestamp),
timestamp)
)

Really Long URLs
• Making graph but the URL is so long
browsers are clipping them?
• Send query string data as a POST

Client Side
Rendering
• yeah...
• works ok with a small number of points
• crashes existing browsers with large
number of points
• Server side faster in many cases!
• We'll try again in 2014

Colors and
ChartJunk
• Default color scheme is gross
• Be kind to the handicapped (uhh, me)
http://colorbrewer2.org/
• Good overview here:
http://bit.ly/10Hu7zU

Looking for
something to do?

Accelerate with PyPy
• JIT for Python
• ~ 5.9x performance improvement
• Actually works and is stable
• Compatible with twisted and Django

Accelerate with numpy
• numpy provides fast vector
manipulation (C code)
• graphite web gui does a lot of vector
manipulation
• hmmmm.....

Ceres Storage Engine
• "Eventually Fixed Size" storage
• More space eﬃcient == more
performance
• see
http://blog.sweetiq.com/2013/01/
using-ceres-as-the-back-end-
database-to-graphite/

OpenTSB
• Not Graphite, but similar in spirit
• Has "collectors" for basic ops stats
• Used by StumbleUpon, Box.net,
pintrest
• Good: Stores data in HBASE/Hadoop
• Bad: Stores data in HBASE/Hadoop

Add More Functions
• coursen (I'm looking at you Ian Malpass,
that's useful for client-side rendering)
• Real vertical lines (our hacks are stupid)
• Better operators (would nice to know
easily how many metrics you have, e.g.
select count(*))

Mine the Apache Log
• Which stats are used the most?
• What are really slow queries?
• Can you optimize them?
• What time frames are used?
• How much old data do you really need
to store?
it's in the
query string

Add a TinyURL
Feature
• The URLs get really long and are hard
to put into email, etc.
• Make a tinyurl feature into the django
app and integrate into dashboard.

Write Docs
yeah you!

Nick Galbreath
http://www.client9.com/
nickg@client9.com
http://www.iponweb.com/
ngalbreath@iponweb.net
Lets Make
Some Graphs!

Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013

More Related Content

What's hot (20)

Similar to Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013 (20)

More from Nick Galbreath (17)

Recently uploaded (20)

Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013