Anomaly Detection - New York Machine Learning

© 2014 MapR Technologies 1
Anomaly Detection
How to Find What You Didn’t
Know to Look For
© MapR Technologies, confidential
October 14, 2014

Anomaly Detection:
How To Find What You Didn’t Know to Look For
Ted Dunning, Chief Applications Architect MapR Technologies
Email tdunning@mapr.com tdunning@apache.org
Twitter @Ted_Dunning
Ellen Friedman, Consultant and Commentator
Email ellenf@apache.org
Twitter @Ellen_Friedman

A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)
e-book available courtesy of MapR
http://bit.ly/1jQ9QuL

Practical Machine Learning series (O’Reilly)
• Machine learning is becoming mainstream
• Need pragmatic approaches that take into account real world
business settings:
– Time to value
– Limited resources
– Availability of data
– Expertise and cost of team to develop and to maintain system
• Look for approaches with big benefits for the effort expended

Anomaly Detection

Who Needs Anomaly Detection?
Utility providers using
smart meters

Feedback from
manufacturing assembly
lines

Monitoring data traffic on
communication networks

What is Anomaly Detection?
• The goal is to discover rare events
– especially those that shouldn’t have happened
• Find a problem before other people see it
– especially before it causes a problem for customers
• Why is this a challenge?
– I don’t know what an anomaly looks like (yet)

Spot the Anomaly

Spot the Anomaly
Looks pretty
anomalous
to me

Spot the Anomaly
Will the real anomaly
please stand up?

Basic idea:
Find “normal” first

Steps in Anomaly Detection
• Build a model: Collect and process data for training a model
• Use the machine learning model to determine what is the normal
pattern
• Decide how far away from this normal pattern you’ll consider to
be anomalous
• Use the AD model to detect anomalies in new data
– Methods such as clustering for discovery can be helpful

How hard is it to set an alert for anomalies?
Grey data is from normal events; x’s are anomalies.
Where would you set the threshold?

Basic idea:
Set adaptive thresholds

What Are We Really Doing
• We want action when something breaks
(dies/falls over/otherwise gets in trouble)
• But action is expensive
• So we don’t want too many false alarms
• And we don’t want too many false negatives
• What’s the right threshold to set for alerts?
– We need to trade off costs

A Second Look

A Second Look
99.9%-ile

New algorithm: t-digest

How Hard Can it Be?
Online
Summarizer
x > t ? Alarm !
99.9%-ile
t
x

Detecting Anomalies in Sporadic Events
0.0 0.2 0.4 0.6 0.8 1.0
0 5000 10000 15000 20000
pnorm(centroids[order(centroids)])
counts[order(centroids)]

Using t-Digest
• Apache Mahout uses t-digest as an on-line percentile estimator
– very high accuracy for extreme tails
– new in version Mahout v 0.9
• t-digest also available elsewhere
– in streamlib (open source library on github)
– standalone (github and Maven Central)
• What’s the big deal with anomaly detection?
• This looks like a solved problem

Already Done? Etsy Skyline?

What About This?
0 5 10 15
offset + noise + pulse1 + pulse2
−2 0 2 4 6 8 10
A
B

Model Delta Anomaly Detection
+ δ
Online
Summarizer
δ > t ?
t
99.9%-ile
Alarm !
Model
-

The Real Inside Scoop
• The model-delta anomaly detector is really just a sum of random
variables
– the model we know about already
– and a normally distributed error
• The output (delta) is (roughly) the log probability of the sum
distribution (really δ2)
• Thinking about probability distributions is good
• But how do you handle AD in systems with sporadic events?

Spot the Anomaly
Anomaly?

Maybe not!

Where’s Waldo?
This is the real
anomaly

Normal Isn’t Just Normal
• What we want is a model of what is normal
• What doesn’t fit the model is the anomaly
• For simple signals, the model can be simple …
x ~ N(0,e )
• The real world is rarely so accommodating

We Do Windows

Windows on the World
• The set of windowed signals is a nice model of our original signal
• Clustering can find the prototypes
– Fancier techniques available using sparse coding
• The result is a dictionary of shapes
• New signals can be encoded by shifting, scaling and adding
shapes from the dictionary

Most Common Shapes (for EKG)

< 1 bit / sample
Reconstructed signal
Original
signal
Reconstructed
signal
Reconstruction
error

An Anomaly
Original technique for finding
1-d anomaly works against
reconstruction error

Close-up of anomaly
Not what you want your
heart to do.
And not what the model
expects it to do.

A Different Kind of Anomaly

+ δ
Online
Summarizer
δ > t ?
t
99.9%-ile
Alarm !
Model
-

The Real Inside Scoop
• The model-delta anomaly detector is really just a sum of random
variables
– the model we know about already
– and a normally distributed error
• The output (delta) is (roughly) the log probability of the sum
distribution (really δ2)
• Thinking about probability distributions is good

Anomalies among sporadic events

Sporadic Web Traffic to an e-Business Site
It’s important to know if traffic is stopped or
delayed because of a problem…
But visits to site normally come at
varying intervals.
How long after the last event
should you begin to worry?

Sporadic Web Traffic to an e-Business Site
It’s important to know if traffic is stopped or
delayed because of a problem…
But visits to site normally come at
varying intervals.
And how do you let your CEO
sleep through the night?

Basic idea:
Time interval between events is how
to convert to something useful you
can measure

Sporadic Events: Finding Normal and Anomalous Patterns
• Time between intervals is much more usable than absolute times
• Counts don’t link as directly to probability models
• Time interval is log ρ
• This is a big deal

Event Stream (timing)
• Events of various types arrive at irregular intervals
– we can assume Poisson distribution
• The key question is whether frequency has changed relative to
expected values
– This shows up as a change in interval
• Want alert as soon as possible

Converting Event Times to Anomaly
99.9%-ile
99.99%-ile

But in the real world, event
rates often change

Time Intervals Are Key to Modeling Sporadic Events

Model-Scaled Intervals Solve the Problem

+ δ
Online
Summarizer
δ > t ?
t
99.9%-ile
Alarm !
Model
-
log p

Incoming
events
99.97%-ile
Alarm
Δn
Rate
predictor
Rate
history
t-digest
δ> t
t i δ λ(t i - t i - n)
λ
t

Incoming
events
99.97%-ile
Alarm
Δn
Rate
predictor
Rate
history
t-digest
δ> t
λ
t

Slipped Week: Simple Rate Predictor
Nov 02 Nov 07 Nov 12 Nov 17 Nov 22 Nov 27 Dec 02
0 100 200 300 400 500
Main Page Traffic
Date
Hits (x 1000)
A B C D

Poisson Distribution
• Time between events is exponentially distributed
Dt ~ le-lt
• This means that long delays are exponentially rare
P(Dt > T) = e-lT
-logP(Dt > T) = lT
• If we know λ we can select a good threshold
– or we can pick a threshold empirically

Seasonality Poses a Challenge
Nov 17 Nov 27 Dec 07 Dec 17 Dec 27
0 2 4 6 8
Christmas Traffic
Date
Hits / 1000

Something more is needed …
Nov 17 Nov 27 Dec 07 Dec 17 Dec 27
0 2 4 6 8
Christmas Traffic
Date
Hits / 1000

We need a better rate predictor…
Incoming
events
99.97%-ile
Alarm
Δn
Rate
predictor
Rate
history
t-digest
δ> t
λ
t

A New Rate Predictor for Sporadic Events

Improved Prediction with Adaptive Modeling
Dec 17 Dec 19 Dec 21 Dec 23 Dec 25 Dec 27 Dec 29
0 2 4 6 8
Christmas Prediction
Date
Hits (x 1000)

Anomaly Detection + Classification  Useful Pair
• Use the AD model to detect anomalies in new data
– Methods such as clustering for discovery can be helpful
• Once you have well-defined models in your system, you may
also want to use classification to tag those
• Continue to use the AD model to find new anomalies

Recap (out of order)
• Anomaly detection is best done with a probability model
• -log p is a good way to convert to anomaly measure
• Adaptive quantile estimation (t-digest) works for auto-setting
thresholds

Recap
• Different systems require different models
• Continuous time-series
– sparse coding to build signal model
• Events in time
– rate model base on variable rate Poisson
– segregated rate model
• Events with labels
– language modeling
– hidden Markov models

Why Use Anomaly Detection?

Keep in mind…
• Model normal, then find
anomalies
-
• t-digest for adaptive threshold
• Probabilistic models for
complex patterns
0 5 10 15
−2 0 2 4 6 8 10
offset + noise + pulse1 + pulse2
A
B

Dec 17 Dec 19 Dec 21 Dec 23 Dec 25 Dec 27 Dec 29
0 2 4 6 8
Christmas Prediction
Date
Hits (x 1000)
Keep in mind…
• Time intervals are key for
sporadic events
• Complex time shift to predict
rate with seasonality
• Sequence of events reveals
phishing attack

A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)
e-book available courtesy of MapR
http://bit.ly/1jQ9QuL

Thank you for coming today!

Sandbox

Anomaly Detection - New York Machine Learning

More Related Content

What's hot (20)

Similar to Anomaly Detection - New York Machine Learning (20)

More from Ted Dunning (8)

Recently uploaded (20)

Anomaly Detection - New York Machine Learning

Editor's Notes