![Sample Log
IP [TIME] [Method Path] StatusCode ResponseLength [[UserAgent]] ResponseTime](https://image.slidesharecdn.com/anomalydetection-210715172723/85/Web-Crawler-Detection-Model-4-320.jpg)
Web Crawler Detection Model
- 1. By Ahmad Etezadi & Matin Zivdar Log Anomaly Detection
- 2. ● Anomaly detection ● Server Logs Dataset ● Pre Processing ● Models ● Evaluation ● API Summary
- 3. Anomaly detection Anomaly detection is one of the most popular machine learning techniques. In this project, we are asked to identify abnormal behaviors in a system, which relies on the analysis of logs collected in real-time from the log aggregation systems of an enterprise.
- 4. Sample Log IP [TIME] [Method Path] StatusCode ResponseLength [[UserAgent]] ResponseTime
- 5. Pre Processing Feature Extraction Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis sit amet odio vel purus bibendum luctus. Generation of features from data that are in a format that is difficult to analyse directly. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis sit amet odio vel purus bibendum luctus. Feature Transformation Transformation of Data: Encoding, Normalization, etc.
- 7. Extraction URL Depth Iran Network Traffic Statistics OS Family Device Brand http://members.tehran-ix.ir/statistics/ixp/pkts
- 9. Transformation Bucketing Normalization One hot encoding
- 10. ● Learning Algorithm Unsupervised Learning 02 ● K-means ● PCA ● Isolation Forest ● AutoEncoder Supervised Learning 01 ● k-NN ● Local Outlier Factor (LOF) ● SVM ● Neural Networks Based Anomaly Detection Algorithm
- 11. PCA is an unsupervised machine learning algorithm that attempts to reduce the dimensionality. Using PCA, you can reduce the dimensionality data and reconstruct the it. Since anomaly show the largest reconstruction error, abnormalities can be found based on the error between the original data and the reconstructed data. PCA
- 12. Anomaly Samples
- 13. Conclusion
- 14. Isolation forest works on the principle of the decision tree algorithm. Due to the fact that anomalies require fewer random partitions than normal normal data points in the data set. So anomalies are points that have a shorter path in the tree. Isolation Forest
- 15. Anomaly Samples
- 16. Conclusion
- 17. Comparison Isolation Forest vs. PCA Number of Isolation Forest outlier: 124,091 Number of PCA outlier: 123,695 common data in PCA & Isolation Forest outlier: 20,512
- 18. Let's Dig Deeper
- 19. An autoencoder is a special type of neural network that copies the input values to the output values. It does not require the target variable like the conventional Y, thus it is categorized as unsupervised learning. Autoencoder
- 20. Sample of error reconstruction
- 21. Histogram of error by Autoencoder
- 22. Evaluation Models Autoencoder PCA Isolation Forest Follow the Rules! - User agents - Malicious IPs dataset
- 23. Model ROC Curve
- 26. API Example
- 27. We also tried: - Extracting time series features - Labeling all other requests for an anomaly “IP” and “User Agent” as an anomaly request
- 28. Thank You!