Using Queryable State for Fun and Profit

© 2022 New Relic, Inc. All rights reserved.
Using Queryable State
for Fun and Profit
Ron Crocker
New Relic Fellow & Lead Architect, Observability
rcrocker@newrelic.com
@RonCrocker
#FlinkForward

Safe Harbor
This presentation and the information herein (including any information that may be incorporated by reference) is provided for
informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc.
(“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby
is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a
historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future
events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the
statement will include words such as “believes,” “anticipates,” “expects” or words of similar import.
Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date
hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties
transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement
or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations,
and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual
results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further
information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from
time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the
SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New
Relic makes no warranties, expressed or implied, in this presentation or otherwise, with respect to the information provided.

Agenda
01 Problem Context
02 Fun
03 Profit
04 Conclusions

© 2022 New Relic, Inc. All rights reserved
Problem Context
Looking at all the things is
hard

Some New Relic terms
Things (or groups of
things) that report
telemetry to NR
● Unique identifier
● Groupable by type
Entity / Entity type Cell*
Internal analytic DB,
federated among the
telemetry ingest cells
Telemetry data lives
at rest in NRDB
NRDB
A deployment unit of
parts of the NR
system.
Independent units of
capacity, similarly
shaped within a cell
type
* http://highscalability.com/blog/2012/5/9/cell-architectures.html

What is Lookout?
New Relic Lookout provides visibility across your entire digital estate,
highlighting any abnormal signals by looking at the golden metrics.

Golden Metrics
Applications
ElastiCache
3-6 key summarizing metrics per entity type
Google SRE Golden Signals,
extended to everything
swapUsageBytes:
title: Swap usage (bytes)
unit: BYTES
queries:
aws:
select: average(aws.elasticache.SwapUsage.byRedisCluster)
from: Metric
eventId: entity.guid
eventName: entity.name

What’s behind a Lookout query
Three steps
1. Fetch the entities
2. Fetch the time
series data for
each entity &
metric
3. Process each time
series and return
the result

Strategy 1: Direct NRDB Query
Low code solution
✔Few moving parts
✔Supports all time
ranges
✘O(n) query duration

Strategy 2: Look-aside cache, in Redis
Help us Redis,
you’re our only hope
✔Queries change from
O(n) to O(1)
✘Several moving parts
✘90 minute cache
implies valid only for
prior 30 minutes
○prior 90 minutes
for mini-overviews

Golden Metric Buffer system, Redis
edition
●Filter service
forwards Golden
Metrics to Kafka
●Golden-signal-writer
publishes Golden
Metrics to Redis
●Mirror forwards
Golden Metrics to
centralized services
(“watcher”)
ETL FTW

Fun
Let’s welcome Flink and
Queryable State to the
party

Queryable State?
Accesses state inside
the stream processing
job
Available since 1.2.0
Image from https://www.ververica.com/blog/queryable-state-use-case-demo

How does Queryable State work?

Strategy 3: Look-aside cache, in Flink
Flink Queryable State
powers a new way
✔Queries remain O(1)
Fewer moving parts
✘90 minute cache
implies valid only for
prior 30 minutes
○prior 90 minutes
for mini-overviews

Golden Metric Buffer system, Flink edition
●Filter & mirror same
as prior system
●Flink cluster running
live state job
●Query service
bridges query to
Queryable State
client
Fewer JVMs

Flink job graph

Learning 1:
Important choices for Queryable State
Data Key IS
Query Key
Data state
IS
Query
response

Learning 2:
Embrace SessionWindow

Learning 3:
Sliding window without sliding window
Append-then-
rightsize
It’s ok to use
more state than
the queryable
state

Profit
Comparing solution costs

Lookaside cache requirements
Memory-resident
data has minimal
query time
All data in memory
System must survive
a node failure
All data is durable
At least 20% free
capacity
Growable without
compromising “in
memory” or
“durability”
requirements
Growth headroom

Golden Metric Buffer system, Redis edition

Buffer
ElastiCache Redis
(6x cache.r6g.4xlarge)
Buffer Writer
EC2
(1x m5.4xlarge)
7.2 0.6
7.8
Cost estimate, in US$K/mo
Classic Solution
(6x cache.r6g.4xlarge
+
1x m5.4xlarge)
Estimates using AWS Pricing Calculator
(on-demand strategy)
(1 cell)

Buffer
ElastiCache Redis
(162x cache.r6g.4xlarge)
Buffer Writer
EC2
(27x m5.4xlarge)
194.4 16.2
210.6
Classic Solution
+
27x m5.4xlarge)
(27 cells)

Golden Metric Buffer system, Flink edition

There are a lot of Golden Metrics

Store only what’s necessary for the query
Golden Metric kind Submetric monoid
min <double, min(), Double.MAX_VALUE>
max <double, max(), -Double.MAX_VALUE>
sum <double, sum(), 0.0>
latest <double, latest(), 0.0>
count <long, sum(), 0>
average <<long,double>, <sum(),sum()>, <0,0.0>>
percentile <byte[], Distribution.merge(), Distribution.empty()>
uniqueCount <byte[], UniqueCount.merge(), UniqueCount.empty()>

Store only what’s necessary for the query

Flink solution
EC2
(3x r6i.8xlarge)
4.4
16 TM Flink cluster

Cost estimate comparison, in US$K/mo
Flink solution
EC2
(3x r6i.8xlarge)
Classic Solution
+
27x m5.4xlarge)
-206.2
-97%
210.6 4.4

Conclusions
What did we learn today?

We replaced a Redis, but…
… this is not a replacement for Redis

Query drives decisions
when using
Queryable State
You can’t spell Queryable State
without Query

Queryable State has real and tangible value
and
deserves rescue from deprecation
Help me save Queryable State

Thank you.
rcrocker@newrelic.com
@RonCrocker

Using Queryable State for Fun and Profit

More Related Content

What's hot (20)

Similar to Using Queryable State for Fun and Profit (20)

More from Flink Forward (14)

Recently uploaded (20)

Using Queryable State for Fun and Profit