Timeseries - data visualization in Grafana

1
Timeseries - wizualizacja danych w Grafanie.

2
About me
About me:
Marek Szymeczko
Software Developer at msales Poland
Automation & Security Enthusiast
Contact:
marek.szymeczko@msales.com
szymeczkomarek@gmail.com

3
Agenda
Why even use of graphic logs?
InfluxDB
• purpose,
• pros & cons.
Telegraf
• statsd
Grafana
• purpose,
• pros & cons,
• golang code example

4
Why use Grafana?
Quality Assurance

5
Why use Grafana?
Anomaly Detection

8
Monitoring process
Application StatsD Telegraf
Grafana InfluxDB

InfluxDB - introduction and purpose.
Written in Go without external
dependencies.
Time series, metrics and analytics
SQL like database system.
Contains a lot of already built-in
features like:
• aggregations,
• retention policies,
• alerting.
events,
visits,
exceptions,
MEASUREMENTs,
things that happen in time.

InfluxDB - pros.
Very high performance (for a valid job),
HTTPs interface for reading and writing the data,
SQL Syntax,
Built-in downsampling the data,
Continuous Queries,
Retention Policies.

InfluxDB - cons.
no updates, only overwrites
not a magical thing
performance issues when used not for valid job
Load
Writes per
second
Queries per
second
Unique series Req. CPUs Req. Memory
Low < 5k < 5 < 100k 2 - 4 2 - 4 GB
Medium < 250k < 25 < 1kk 4 - 6 8 - 32GB
High > 250k > 25 > 1kk 8+ 32+GB
Infeasible > 750k > 100 > 10kk probably
impossible
probably
impossible

https://github.com/msales/ocoderfest-monitoring
TIG Docker Stack

curl -i -XPOST ‘http://influxdb.ocoderfest.msales:
8888/write?db=metrics' --data-binary
‘page_visit,host=server01 value=1
1503987933000000000'
InfluxDB - basic queries
SHELLHTTP API
curl -i -XPOST http://influxdb.ocoderfest.msales:
8888/query --data-urlencode "q=CREATE DATABASE
metrics”
> CREATE DATABASE metrics
> INSERT page_visit,host=server01 value=1
1503987933000000000
http://influxdb.ocoderfest.msales:8888 docker exec -it $(docker ps -aqf “name=influxdb.container”) influx
> SHOW MEASUREMENTS
curl -i -XPOST http://influxdb.ocoderfest.msales:
8888/query --data-urlencode "q=SHOW MEASUREMENTS”

InfluxDB - structure of data point
key
coffeemaker,location=katowice,machineid=1 water=90,coffee=55 1486738421000000000
tags
values
timestamp

key
name of the measurements
"something like” the mysql table
can be dynamically created

tags
are present after key first comma
"something like” string indexed column in mysql
use tags if they will be commonly-queried meta data
use tags if you plan to use them with group by
tag values are always interpreted as strings

values
are present after key first space
"something like” number non-indexed column in mysql
better not be used with group by
are interpreted as numeric

by default nanosecond precision
“something like” mysql ID
there cannot be more entries with same timestamp
timestamp

InfluxDB - downsampling
name: page_visits
------------
time host value
2017-09-12T00:00:00Z foo.msales.com 271
2017-09-12T00:00:00Z bar.msales.com 233
2017-09-12T00:00:05Z foo.msales.com 212
2017-09-12T00:00:05Z bar.msales.com 312
2017-09-12T00:00:10Z foo.msales.com 237
2017-09-12T00:00:10Z bar.msales.com 162
Many hosts receive
many visits over
short period of time.
name: page_visits_cq_1m
------------
time host value
2017-09-12T00:00:00Z foo.msales.com 17423
2017-09-12T00:00:00Z bar.msales.com 19483
2017-09-12T00:01:00Z foo.msales.com 17329
2017-09-12T00:01:00Z bar.msales.com 19721
We can group
historical data
without losing
“much” information.
5s interval
1m interval

InfluxDB - retention policies and continuous queries
CREATE CONTINUOUS QUERY “cq_1m_visits" ON “metrics"
RESAMPLE FOR 2m
BEGIN
SELECT sum(value) AS visits
INTO "_internal"."page_visits".page_visits_cq_1m
FROM page_visits
GROUP BY time(1m), host
END
CREATE RETENTION POLICY "page_visits" ON “metrics" DURATION 30d
REPLICATION 1 DEFAULT
SHOW RETENTION POLICIES ON “metrics”

21
Monitoring process
Grafana InfluxDB

22
Telegraf
Written in Go.
Made by same company that developed InfluxDB.
Plugin driven concept:
• input plugins - collect metrics from system or apps (nginx, rabbitmq, statsd)
• processor plugins - decorate and prepare the data,
• aggregate plugins - create aggregate metrics (mean, max, percentiles)
• output plugins - writes to various destinations (influxdb, graphite, prometheus)

23
Telegraf
Processors plugins process metrics as
they pass and emit results based on the
values:
• ex. add tag to all metrics that pased.
Aggregator plugins are more complex,
they emit new aggregate metrics such
as:
• mean, min, max, quantile.
They need to be ran with period, so it
means they aggregate values of the
past: (now() - period)

24
Monitoring process
Grafana InfluxDB

Statsd
statsd - Telegraf input plugin that works on UDP (fast, fire and forget) port,
let’s us parses the messages and extracts metrics data,
periodically flushes the data,
client has a really light package, available for all modern languages:
• php: https://github.com/domnikl/statsd-php
• go: https://github.com/cactus/go-statsd-client
• nodejs: https://github.com/msiebuhr/node-statsd-client

Statsd - context
func (s Statsd) Inc(name string, value int64, rate float32, tags map[string]string) error {
name += formatTags(tags)
return s.client.Inc(name, value, rate)
}
Go ahead and use influxdb tags!

Statsd - metrics
func main() {
statsd, err := stats.NewStatsd("127.0.0.1:8125", "go")
if err != nil {
return;
}
}
connect to statsd deamon
only place when statsd can throw exception

func main() {
...
statsd.Inc("page_visit", 1, 1.0, nil)
}
Statsd - increment metric.
Translates to:
Image source: http://influxdb.ocoderfest.msales:8888

func main() {
...
tags := make(map[string]string)
tags["browser"] = "chrome"
statsd.Inc("page_visit", 1, 1.0, tags)
statsd.Dec(“page_visit", 1, 1.0, tags)
}
Statsd - periodically flushed data.
Translates to:

func main() {
...
stamp := time.Now()
time.Sleep(time.Second * 5)
statsd.Timing("conversion_time", time.Since(stamp), 1.0, nil)
}
Statsd - timing metric.
Translates to:

func main() {
...
stamp := time.Now()
for j := 0; j < 10; j++ {
go func(client stats.Stats, j int, stamp time.Time) {
time.Sleep(time.Second * time.Duration(j))
client.Timing("conversion_time", time.Since(stamp), 1.0, nil)
}(client, j, stamp)
}
}
Statsd - timing metric.
Translates to:

func main() {
...
statsd.Gauge("happy_clients", 1, 1.0, nil)
}
Statsd - gauge metric.
Translates to:

33
Monitoring process
Grafana InfluxDB

Grafana - introduction and purpose.
Open Sourced,
Suite used for data and applications
analytics and visualization,
Accepts multiple data sources,
“Query language and capabilities of
each Data Source are obviously very
different.”
Possible data sources:
Graphite,
InfluxDB,
OpenTSDB,
Prometheus,
Elasticsearch,
CloudWatch,
MySQL.

Grafana - pros.
free,
permission system to maintain user and team access,
multiple data sources,
easy, beautiful, fast and interactive,
built-in alerts.

Grafana - cons.
no built-in machine learning for alerts,
no search or data exploring,
data source defines the way panel must be created.

Grafana - structure
One or more (usually one) organization.
Each organization can have one or more data sources.
All dashboards are owned by perticular organization.
Dashboards contains rows that contain single panels (which might use different
data sources).
Single Panel uses query editor to query the data source for metrics to display on it.
Users can have access to dashboards, or organizations within many different
permissions.

https://github.com/m4r3x/go-event-simulation
TIG Stack Simulation + Dashboard Export

41
Thanks.
msales Poland
Sobieskiego 2, 40-082 Katowice
Poland
follow us @msalestech
info@msales.com www.msales.com +48 32 630 40 76

Timeseries - data visualization in Grafana

More Related Content

What's hot (20)

Similar to Timeseries - data visualization in Grafana (20)

Recently uploaded (20)

Timeseries - data visualization in Grafana