Why You Should NOT Be Using an RDBMS for Time-stamped Data

Thom Crowe
Community Manager
InfluxData
Why You Should NOT Be
Using an RDBMS for Time-
Stamped Data

© 2019 InfluxData. All rights reserved.2
Why Time-
Series for
Monitoring,
Metrics, Real-
Time Analytics
and IoT/Sensor
Data
● What is time series data
● Differences between Time Series
Databases (TSDBs)
● InfluxDB Data model

What is time series data?

Time series data is
made from the same source over a time interval.
Plot the points on a
graph and one of
your axes would
always be time.
a sequence of data points,
typically consisting of successive measurements

Time Series data is...

Not Time Series data...

Time Series data...

Studen
t
Grades
Time spent studying

© 2019 InfluxData. All rights reserved.14 © 2019 InfluxData. All rights reserved.14
Regular vs Irregular Time Series
Metrics (Regular)
Events (Irregular)
Measurements
gathered at
regular time
intervals
Measurements
gathered at irregular
time intervals

Regular Time Series Irregular Time Series
Measurements
gathered at regular
time intervals
Measurements
gathered at irregular
time intervals
Metrics Events

Summarization of Events
Events become regular time intervals, for example
Summarizing the average
trade price of Apple stock
every 10 minutes over the
course of a day
Summarizing the average
response time for
requests in an application
over 1 minute intervals

What is a time series database?

Characteristics
of the Data
• All Time-stamped data
• Generated in regular (Metric) and
irregular (Event) time periods
• Huge volumes of data
• Real-time and time sensitive

Time Series
databases are
optimized for
collecting, storing,
retrieving &
processing of Time
Series data
Compare this to
• Document databases
Optimized for storing JSON documents
• Search databases
Optimized for full-text searches
• Traditional relational
Databases optimized for the tabular
storage of related data in rows & columns

Some Databases used for Time Series
Other DB
Types
Time Series
DBs

DB-Engines Results
Source: DB-Engines Source: DB-Engines
~138% increase in DB-Engines score over 24 months
Trend of the Last 24 Months
CATEGORY/BRAND Q1

Time-series use cases

Primary Use Cases
Custom monitoring
solutions to track
servers, VMs, applications,
users or events
Industrial settings:
factories, oil & gas,
agriculture, smart roads &
infrastructure
Consumer: wearables,
consumer devices & trackers
Apps that instrument
business, social or
development metrics in
real-time
IoT DevOps Real-Time
Analytics

Platform Strategy:
Be The Platform of Choice for All Metrics and Event Workloads
Common Metrics
and Events
Platform
Application,
Custom Logs &
Traces
Business
Metrics
Infrastructure
& Application
Metrics
IoT
Sensor
Events

InfluxDB Platform Features
INSTRUMENT OBSERVE
AUTOMATE LEARN
✓ Quickly ingest data from
everywhere
✓ Efficiently store (Compress)
the data at scale
✓ Support real-time query,
analysis and visualization of
large data sets
✓ Provide time-based functions
for “change over time”
analysis and control
✓ Provide automation and
control functions
✓ Evict and down-sample data
✓ Facilitate machine learning
and anomaly detection
algorithms
✓ Provide streaming analytics
for data in motion
METRICS
EVENTS

InfluxDB Products and Offerings

Why Choose
InfluxDB
• Easy to get started with
• Familiar query syntax
• No external dependencies
• Allows for regular and irregular time
series
• Horizontally scalable
• Member of a cohesive time series
platform

InfluxDB Data Model

A typical time series graph

The Label
We call this the measurement

The Legend (metadata)
We call these tags. Tags are indexed.

Collection of all tags
We call this the tagset.
ticker=A,market=NASDAQ
ticker=AA,market=NYSE
ticker=AAPL,market=NASDAQ

Y-Axis Values
We call these fields. Note that the values that the field stores can be
floats, ints, strings, or bools.

The Collection of Fields
We call this the fieldset. Note that in this case, there's only one
field. (There could be many.)
price=177.03
price=32.10
price=35.52

X-Axis Value
We call this the timestamp.

How do we represent points
textually?

Points in InfluxDB look like…
stock_price,ticker=A,market=NASDAQ price=177.03 1445299200000000000
stock_price,ticker=AA,market=NYSE price=32.10 1445299200000000000
stock_price,ticker=AAPL,market=NASDAQ price=45 1445299200000000000a

The Line protocol
measurement,tagset fieldset timestamp
stock_price,ticker=A,market=NASDAQ price=177.03
14452992000000000

Concepts: Time Series Database Schema
Data Ingestion Format
• Points are written to InfluxDB using the Line Protocol, which
follows the following format:
<measurement>[,<tag-key>=<tag-value>] [<field-key>=<field-value>]
[unix-nano-timestamp]
cpu_load,hostname=server02,az=us_west temp=24.5,volts=7
1234567890000000
Reference: https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_reference/
Measurement Tag Set Field Set
Timestamp

A Series in InfluxDB
measurement + tagset = the series as a whole
measurement + tagset + timestamp = single point

Examples of points in Line Protocol
cpu,host=server1 value=100 1445299200000000000
temperature,zipcode=94107,country=usa value=75,humidity=10
1445299200000000000
response_time,method=GET,precision=ms value=12i
1445299200000000000

Use Cases

“We analyze over 70,000 hours of
data every night, half a billion data
points, to produce alerts for our
technicians. Having this real-time
data in the cloud makes it
possible to identify trends, usage
patterns & even detect problems
before they exist!”
David McLean
Lead Developer at BBOXX

Wayfair is using InfluxData to
monitor system metrics & events
across datacenters, and Real
User monitoring (RUM) to
understand user experience on
their e-commerce site. The goal
is to marry these with business
process events and provide
better business insight and
competitive advantage

• InfluxDB is a major component in Wayfair’s Cyber 5 Holiday Weekend monitoring
and alerting systems
– Data center metrics: 100s of apps sending metrics from multiple data
centers
– Real User Monitoring (Client side monitoring)
• Understand user experience - deploy 100s of code changes to the app, each
change has the potential to impact performance for better or worse
– Daily, 20 million RUM measurements across 8 stores, hundreds of page
types, & thousands of device types (phones, tablets, laptops & PCs)
• Wayfair is working strategically with InfluxData
– Ensure the Wayfair implementation is scalable, robust, and in line with the
future direction of InfluxDB
– Providing case studies to help drive InfluxDB enterprise feature requests
Measuring performance of a high-traffic
e-commerce site

Architecture
• Multi-layer pipeline with Telegraf
– Receive raw InfluxDB line protocol from
applications via UDP and forward it on to Kafka
– Consumes metrics from the Kafka buffer and
writes to InfluxDB
• Connects multiple data centers by mirroring Kafka
topics to shuttle metrics (vs through cross-datacenter
db replication)
• Ability to use UDP (fast, non-blocking) and TCP (more
transactionally robust)
• Ability to inject various processing hooks into the data
stream as the business needs evolve
• Easy to write the same data to multiple instances of
InfluxDB
• Multi-day tolerance against a severe network
connectivity incident

Graphite limitations
✓ No clustering
✓ Storage infrastructure that is
difficult to manage
✓ No out-of-the-box data pipeline
solution
✓ Graphite maxed out its
performance potential
✓ No shard relocation &
backup/restore
InfluxDB Advantages
✓ Rapid performance improvements &
hardening
✓ Supports true clustering
✓ Written in Go, a language that fully
exploits multi-core environments
✓ Built with an emphasis on efficient &
scalable storage
✓ Active community & an ecosystem
of related tools for building
enterprise-wide installations
✓ Growing customer base from a
diverse set of companies
Why replace Graphite w/ InfluxDB

InfluxDB is a major component to
providing observability across the
board at Playtech. It is better to
have simpler and faster
monitoring systems with tools for
post hoc analysis and avoid
“magic” systems that try to
automatically detect causality.
Aleksandr Tavgen
Technical Archtiect, Playtech

• 50+ multi-branded sites, distributed worldwide, multiple products & channels
– Handle financial transactions in strictly regulated markets
– There are also different variations in their back end
• InfluxDB is a major component to providing observability across the board
– Production System level Monitoring
• Low-level Monitoring—CPU, disk, memory usage, networking, garbage
collection cycles, etc. Cannot reflect business logic processes due to
the # of interdependencies between various components and services
• High-level Monitoring—KPIs (user sessions, payments, transaction
amounts, logins, etc.). Possible to indirectly monitor complex system
behavior, especially in the case of distributed systems.
– Organizational level Monitoring
• Map of all product portfolios, clients, sites, & brands tied to internal org
chart (who is responsible for a features, who wrote the code, etc)
Monitoring distributed systems

Architecture
• Service-oriented architecture
where its components have very
complex business logic and
states
• Natural way of defining the
boundaries and responsibilities
of each component; easier to
add features independently and
on-the-fly without the fear of
breaking something
• The microservices are executed
on their own Python VMs
• System has an event-driven
design. All communication goes
through the message queue and
the system works in an
asynchronous way

SHA limitations
✓ Black box – impossible to
change its settings or tune it
✓ Yields a huge amount of false
positives and false negatives
✓ SHA’s granularity is 15
minutes
✓ Hewlett-Packard’s SHA does
not work as expected - trying
to be a universal instrument
InfluxDB Advantages
✓ Time Series DB for any metric
✓ Widely adopted by the community
✓ Fast and reliable
✓ Comes with great support
Why replace HP Service Health
Analyzer (SHA) w/ InfluxDB

A&S Energie is a Belgian biomass
plant that supplies electricity by
burning non-recyclable wood.
They replaced their existing
solution with Factry Historian,
powered by open source
InfluxDB. This allowed A&S
Energie to increase use of the
process data to make their staff
more efficient as well as find
failures faster.

• A&S Energie presented a classic Industrial IoT challenge involving
high-cost, confined industrial equipment requiring a useful life of
many decades.
– Operators wrote down counters on paper (values of certain
measurements needed for reporting on the power plant’s status)
– Based on this paper-reliant reporting, Management reporting was done
with Excel
– The existing Historian was underutilized—its data was not being used to
extract meaning - Historical view only provided 2 weeks’ worth of data
• Factry Data Historian with InfluxDB
– OPC UA Connecter, InfluxDB for storage and Grafana for visualization
Measuring performance of a high-traffic
e-commerce site

Architecture
• Datasources
– From several sensor
sources (Siemens,
Rockwell
Automation,
Schneider Electric)
– Uses an OPC UA
collector
• Stores all time-stamped
values in InfluxDB
• All other contextual data
from multiple sources
• Grafana dashboards
provided to management
and operators

Existing Data Historian
✓ No easy access to long term
data
✓ Still too many disparate
databases including Excel and
paper
✓ Legacy software – Windows
2000
✓ Not accessible – in a closet –
really!
✓ Not easy to expand on the
data
InfluxDB Advantages
✓ Real-time and historical insights for
everyone, from machine operator to
plant manager
✓ Handling high write loads
✓ Platform-independent - Most
historians are based on proprietary
systems and are Windows-based
✓ Retention policy options - keeps the
data at its original resolution for as
long as needed
Why replace their Data Historian

Writing Data into InfluxDB

Creating a database
CREATE DATABASE mydb
Last login: Mon Oct 19 10:50:43 on ttys006
~$ influx
Connected to http://localhost:8086 version 0.9
InfluxDB shell 0.9
> create database mydb
>

Verifying that it was created
SHOW DATABASES
> show databases
name: databases
---------------
name
_internal
mydb

Using the database we just created
USE mydb
> use mydb
Using database mydb
>

Inserting data into the database
insert cpu,host=server1,location=us-west value=10
insert cpu,host=server1,location=london value=11
insert cpu,host=server2,location=us-west value=12

Verifying that the data was written
SELECT * FROM cpu
// PS. Be careful! This query can be very expensive.
SHOW SERIES
SHOW MEASUREMENTS

Why You Should NOT Be Using an RDBMS for Time-stamped Data

More Related Content

What's hot (20)

Similar to Why You Should NOT Be Using an RDBMS for Time-stamped Data (20)

More from DevOps.com (20)

Recently uploaded (20)

Why You Should NOT Be Using an RDBMS for Time-stamped Data