The present and future of serverless observability

and FUTUREThe
Serverless
OBSERVABILITY
ofpresent

available in:
Austria, Switzerland,
Germany, Japan and
Canada
30+ platforms

follow @DAZN_ngnrs
for updates about the
engineering team
We’re hiring! Visit
engineering.dazn.com
to learn more.

http://bit.ly/yubl-serverless

http://bit.ly/2Cdsai5

Monitoring
watching out for
known failure modes
in the system,
e.g. network I/O, CPU,
memory usage, …

Observability
being able to debug
the system, and gain
insights into the
system’s behaviour

http://bit.ly/2EXKEFZ

mm… I wonder what’s
going on here…

These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

microservices death stars circa 2015

I got this!

NOWHERE
to install agents/daemons

•nowhere to install agents/daemons
new challenges

user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
handler
handler
handler
handler
handler
handler
handler

user request
user request
user request
user request
user request
user request
user request
critical paths:
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead

user request
user request
user request
user request
user request
user request
user request
critical paths:
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead
NO background processing
except what platform provides

•no background processing
new challenges

EC2
concurrency used to be
handled by your code

EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, it’s handled by the
AWS Lambda platform

EC2
logs & metrics used to be
batched here

EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, they are batched in each
concurrent execution, at best…

HIGHER concurrency to log
aggregation/telemetry system

•higher concurrency to telemetry system
new challenges

Lambda
data is batched between
invocations

Lambda
idle
data is batched between
invocations

Lambda
idle
garbage collectiondata is batched between
invocations

Lambda
idle
garbage collectiondata is batched between
invocations
HIGH chance of data loss

•high chance of data loss (if batching)
new challenges

my code
send metrics
internet internet
press button something happens

http://bit.ly/2Dpidje

?
functions are often chained together
via asynchronous invocations

?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
tracing ASYNCHRONOUS
invocations through so many
different event sources is difficult

•asynchronous invocations
•high chance of data loss (if batching)
new challenges

2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp Request Id
your log message

one log group per
function
one log stream for each
concurrent invocation

logs are not easily searchable in
CloudWatch Logs
me

CloudWatch Logs AWS Lambda ELK stack

those extra 10-20ms for
sending custom metrics would
compound when you have
microservices and multiple
APIs are called within one slice
of user event

Amazon found every 100ms of latency cost them 1% in sales.
http://bit.ly/2EXPfbA

console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
timestamp metric value
metric type
metric namemetrics
logs

CloudWatch Logs AWS Lambda
ELK stack
logs
m
etrics
CloudWatch

delay
cost
concurrency
no latency
overhead

API Gateway
send custom metrics
asynchronously

SNS KinesisS3API Gateway
…
send custom metrics
asynchronously
send custom metrics as
part of function invocation

narrow focus on a function
good for homing in on performance issues
for a particular function, but offers little to
help you build intuition about how your
system operates as a whole.

However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

don’t span over async
invocations
good for identifying dependencies of a function,
but not good enough for tracing the entire call
chain as user request/data flows through the
system via async event sources.

don’t span over non-AWS services

our tools need to do more to help us with
understanding & debugging our distributed
system, not just what happens inside one function

“one user action/vertical slice through the system”

HELP…

WARNING: this is part fiction, part inspired by new tools

different dimensions of service X
splattered across the screen

+cold starts
+throttled invocations
+concurrent executions
+estimated cost ($)

SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

birds-eye view of our system as it lives and breathes

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
trace async invocations
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
trace non-AWS resources
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
Logs
POST /user2018/01/25 20:51:23.188 incoming request…
level
debug
request-id
start-time
0ae4ba5d-dab1-4f9e-9de7-eace27ebfbc2
2018/01/25 20:51:23.188
method POST
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
Logs
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
create-auth0-user

user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
Logs
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
click here to go to code create-auth0-user

Logs Input/Output
user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
input output
{
"body": "{ "username":"theburningmonk"}",
"resource": "/user",
"requestContext": {
"resourceId": "123456",
"apiId": “1234567890",
"resourcePath": "/user",
{
"statusCode": 200
}
create-auth0-user

Logs Input/Output
user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
input output
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
{
"error": null,
"result": "OK"
}
create-auth0-user

Logs Input/Output
user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
Face API
input error
{ "Records": [
{ "Sns": {
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
create-auth0-user

Logs Input/Output
user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
Face API
input error
{ "Records": [
{ "Sns": {
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
!

All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

Input/Output
user
proﬁle-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

all your needs in one placeTRACING

what if we can query it
like a graph?

http://amzn.to/2nk7uiW

ability to query based on the relationship
between observed components
(as well as the components themselves)

the elevated error rate in service X was caused by
DynamoDB table throttling.“
”

payment was slow last
night around 10PM.
investigate.

time
95-percentile latency
service A
service B
10PM

time
95-percentile latency
service A
service B
10PM
causality? or correlation?

user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
payment-service
DEPENDS_ON
“payment was slow last
night around 10PM”
user-table

user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
DEPENDS_ON
payment-service
user-table
throttled exceptions!

user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

use ML to auto-detect erroneous or
suspicious behaviours, or to suggest
possible improvements

!
Function [X] just performed an
unexpected write against
DynamoDB table [Y].
Should I…
ignore it from now on
shut it down!!

Observability Bot <bot@bestobservability.com>

don’t bother me about this again

auto-modify IAM role with DENY rule

Function [X]’s performance
has degraded since yesterday
- 99% latency has gone up by
47% from 100ms to 147ms.
!

!
Function [X] can run faster &
cheaper if you increase its
memory allocation.
Should I…
ignore it from now on
update setting

zzz… the future of… zzz …
serverless observability… zzz

Simon Wardley
context &
movement

The best way to predict the future
is to invent it.
- Alan Kay

The best way to invent
the future is to inception
someone else to do it.
- me

The present and future of serverless observability

More Related Content

What's hot (15)

Similar to The present and future of serverless observability (20)

More from Yan Cui (20)

Recently uploaded (20)

The present and future of serverless observability