Anomaly Detection by Mean and Standard Deviation (LT at AQ)
Download as PPTX, PDF3 likes4,124 views
The document discusses anomaly detection and motivation for detecting anomalies in system monitoring. It notes that traditional monitoring looks for fixed rules and thresholds but behaviors can vary over time, services, and other factors. The key to anomaly detection is watching for differences between usual and unusual behaviors. Examples are given of applying anomaly detection to network traffic amounts and email spam rates to detect issues like DDoS attacks or misconfigured spam filters. Heuristics are suggested like only alerting if an anomaly persists for over 15 minutes to avoid false alarms from temporary spikes.
Anomaly Detection by Mean and Standard Deviation (LT at AQ)
- 3. Motivation for detecting anomaly Traditional system monitoring • process existence • ping, http, tcp response • disk usage → “fixed” rule / threshold
- 4. Motivation for detecting anomaly Notice something out of ordinary • network traffic is heavier than usual • number of login try is obviously larger • a colleague is strangely gracious today → Unusual behaviors; Indications of fault. Such info helps preventing service degrading in advance!! but rule/threshold vary with service, host, client, time…
- 5. key to detect anomaly usual unusual Watch differences b/w
- 6. e.g. Network Traffic Mon Tue Wed Thu Fri traffic time
- 7. Superimpose 24 hour plot Traffic at 15:00 on workday is about 1.2 Gbps traffic time Periodicity!!
- 8. mean mean － 3σ mean + 3σ amount of dispersion from mean Acceptable “range” → e.g. Acceptable range of traffic at 15:00 on workday is 1.01 to 1.38 Gbps
- 11. number of mail passed spam filterspam rate e-mail Applied a wrong spam rule
- 12. However Reality is not that simple… 人生楽ありゃ苦もあるさ 涙の後には虹も出る 歩いてゆくんだしっかりと 自分の道をふみしめて 山上路夫
- 13. downloading large files mass e-mail sending “Traffic spike” happens so frequently Frequent false-positive alerting will be “cry-wolf” system…
- 14. heuristic filtering In usual, traffic gets cool down within 15 minutes notify engineers if anomaly continues more than 15 minutes Engineers’ knowledge is gold mine for better algorithm → one practical example: