Buzzwords 2014 / Overview / part2
0 likes636 views
This document provides an overview of a talk given by Ted Dunning on anomaly detection in time-series databases. The talk discusses using deep learning models like auto-encoders to learn what normal patterns look like in time-series data and detect anomalies as large errors when reconstructing input signals. It also describes using windowing techniques and clustering common shapes to build a dictionary that can reconstruct signals by combining entries from the dictionary.
Buzzwords 2014 / Overview / part2
- 1. BERLIN BUZZWORDS 2014 SELECTEDTALKS OVERVIEW tech talk @ ferret Andrii Gakhov 19/06/2014
- 2. DEEP LEARNING FOR HIGH PERFORMANCETIME-SERIES DATABASES byTed Dunning
- 3. ABOUT OF THE AUTHOR • Chief Application Architect at MapR Technologies • Ph.D. in computing science from the University of Sheffield • Committer on Mahout, Drill, Zookeeper … • http://tdunning.blogspot.de/ • @ted_dunning
- 5. ANOMALY DETECTION 99.9%-ile Online summarizer (t-digest) 99.9%-ile t x > t? x ! • The t-digest algorithm was developed byTed Dunning and available in Apache Machout • With t-digest algorithm one can accurately estimate quantiles for very large data sets with limited memory use input signal
- 8. NOT SURE
- 9. WHAT IS NORMAL? • We need to have a model of what is normal • Everything that doesn’t fit model is the anomaly • For simple signals we can assume just normal distribution
- 10. WINDOWS
- 11. WINDOWS
- 12. WINDOWS • Set of windowed signals - model of the original signal • Clustering can find the prototypes • The result is a dictionary of shapes • New signals can be encoded by shifting, scaling and adding shapes from the dictionary
- 14. RECONSTRUCTION
- 15. ANOMALY
- 16. ANOMALY
- 17. MODEL ANOMALY DETECTION Online summarizer 99.9%-ile t ∆ > t? x ! t ∆ reconstruction error Model x’ x-x’ input signal
- 18. COMPRESSION Minimal Error Maximum likelihood Maximum compression • Good anomaly detectors give good compression • So, we are constructing an auto-encoder!
- 19. MODEL ANOMALY DETECTION Online summarizer 99.9%-ile t ∆ > t? x ! t ∆ reconstruction error Encoder x’ x-x’ input signal shape dict
- 20. CLUSTERING • Use windowing • Find nearest cluster for each window • Scale cluster to the right size • Subtract from the original signal
- 21. CLUSTERING AS NEURAL NETWORK
- 22. OVERLAPPING NETWORKS Time series input Reconstructed time series
- 23. READ MORE? • A New Look At Anomaly Detection • This is the second book in the series Practical Machine Learning by Ted Dunning & Ellen Friedman • FREE download from www.mapr.com