MLSEV Virtual. Anomaly Detection Examples
0 likes381 views
The document discusses practical applications of anomaly detection using BigML, focusing on three demos: removing outliers, fraud detection, and discovering novel categories. Anomaly detectors are unsupervised algorithms that identify unusual instances in datasets, improving model performance and addressing issues like model degradation in production. The use cases of anomaly detection include fraud detection in financial transactions and identifying unclassified categories in evolving datasets.
MLSEV Virtual. Anomaly Detection Examples
- 1. 2nd edition
- 2. #MLSEV 2 Anomaly Detectors Practical Examples with BigML Guillem Vidal Machine Learning Engineer, BigML
- 3. #MLSEV 3 Outline 2 Demo 1: Removing Outliers 3 Demo 2: Fraud Detection 4 Demo 3: Novel Categories Discovery 1 Anomaly Detection Recap
- 4. #MLSEV 4 Anomaly Detection Recap
- 5. #MLSEV 5 Anomaly Detection date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51 An unsupervised algorithm that looks for unusual instances in a dataset. Anomaly detectors provide an anomaly score to each instance, the higher is the score the most unusual is the instance. Example: • Amount $2,459 is higher than all other transactions • Only transaction • In zip 21350 • For the purchase class “tech"
- 6. #MLSEV 6 Graphical Example Which object appears more unusual within this group?
- 7. #MLSEV “Round”“Skinny” “Corners” “Skinny” but not “smooth” No “Corners” Not “Round” Most unusual 7 Graphical Example
- 8. #MLSEV 8 Isolation Forest “easy” to isolate “hard” to isolate Depth Now repeat the process several times and use average depth to compute anomaly score: 0 (similar) 1 (dissimilar) Isolation Forest: Grow random decision trees until each instance is in its own leaf. Random features and splits
- 9. #MLSEV 9 Isolation Forest Splits https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf AnomalyUsual data point
- 12. #MLSEV 12 Outliers • Data points that differ significantly from other observations • Outliers can cause serious problems in statistical analyses • Examples: 1 2 3 4 5 6 10 20 30 40 50 60 70 80 900 Price (100k €) Square Meters Regression: 1 2 3 4 5 6 0 Price (100k €) 10 20 30 40 50 60 70 80 90 Square Meters Unsold Sold Classification:
- 13. #MLSEV 13 Outliers • Data points that differ significantly from other observations • Outliers can cause serious problems in statistical analyses • Examples: 1 2 3 4 5 6 10 20 30 40 50 60 70 80 900 Price (100k €) Square Meters Regression: 1 2 3 4 5 6 0 Price (100k €) 10 20 30 40 50 60 70 80 90 Square Meters Unsold Sold Classification:
- 14. #MLSEV 14 Removing Outliers ORIGINAL DATASET TRAIN SET TEST SET ALL MODEL CLEAN DATASET REJECT MOST ANOMALOUS CLEAN MODEL COMPARE EVALUATIONS ANOMALY DETECTOR • Anomaly detectors can be used to remove outliers • With this methodology outliers removal can be tested ALL EVALUATION CLEAN EVALUATION
- 15. #MLSEV 15 Outliers Demo pregnancies plasma glucose blood pressure triceps skin thickness insulin bmi diabetes pedigree age diabetes 6 148 72 35 0 33.6 627 50 TRUE 1 85 66 29 0 26.6 351 31 FALSE 8 183 64 0 0 23.3 672 32 TRUE 1 89 66 23 94 28.1 167 21 FALSE 0 137 40 35 168 43.1 2.288 33 TRUE 5 116 74 0 0 25.6 201 30 FALSE 3 78 50 32 88 31.0 248 26 TRUE 10 115 0 0 0 35.3 134 29 FALSE 2 197 70 45 543 30.5 158 53 TRUE 8 125 96 0 0 0.0 232 54 TRUE 4 110 92 0 0 37.6 191 30 FALSE 10 168 74 0 0 38.0 537 34 TRUE Diabetes dataset • Predict whether patients are diabetic or not BigML Gallery
- 16. #MLSEV 16 Summary •An anomaly detector improved a classifier performance by removing top 10 anomalies as outliers •Usually removing anomalies with score over 60% works
- 18. #MLSEV 18 Fraud Detection HISTORIC NON FRAUD TRANSACTIONS ANOMALY DETECTOR NEW TRANSACTION(S) ANOMALY SCORE KEEP HIGH SCORES SUSPICIOUS TRANSACTION(S) FRAUD ANALYST • Use Machine Learning to detect fraudulent financial transactions • Fraud transactions being unusual can be detected with an anomaly detector
- 19. #MLSEV 19 Fraud Detection Demo Credit card transactions dataset • Anonymized credit card transactions with a fraud label • Very unbalanced Time V1 V2 V3 V4 0 -1.3598 -0.0727 2.5363 1.3781 0 1.1918 0.2661 0.1664 0.4481 1 -1.3583 -1.3401 1.7732 0.3797 1 -0.9662 -0.1852 1.7929 -0.8632 2 -1.1582 0.8777 1.5487 0.4030 2 -0.4259 0.9605 1.1411 -0.1682 4 1.2296 0.1410 0.0453 1.2026 7 -0.6442 1.4179 1.0743 -0.4921 7 -0.8942 0.2861 -0.1131 -0.2715 9 -0.3382 1.1195 1.0443 -0.2221 10 1.4490 -1.1763 0.9138 -1.3756 V27 V28 Amount Class 0.1335 -0.0210 149.62 0 -0.0089 0.0147 2.69 0 -0.0553 -0.0597 378.66 0 0.0627 0.0614 123.5 0 0.2194 0.2151 69.99 0 0.2538 0.0810 3.67 0 0.0345 0.0051 4.99 0 -1.2069 -1.0853 40.8 1 0.0117 0.1424 93.2 0 0.2462 0.0830 3.68 0 0.0428 0.0162 7.8 0 … … … https://www.kaggle.com/mlg-ulb/creditcardfraud
- 20. #MLSEV 20 Summary • Anomaly detectors can be an unsupervised alternative to classifiers in extremely unbalanced datasets • Fraud detection is an example. A similar approach can be used for other use cases such as predictive maintenance or network intrusion detection • With this approach, the most challenging aspect is finding the features that work
- 21. #MLSEV 21 Novel Categories Discovery
- 22. #MLSEV 22 Novel Categories • A classification model performance could be reduced over time in production with real data evolution over time • Model degradation can be addressed by retraining with new data • What if new data is not labeled? • What if new data contains novel categories? • Anomaly detectors can be used to spot model degradation and to discover novel categories
- 23. #MLSEV 23 Novel Categories Discovery ORIGINAL DATASET CLASSIFICATION MODEL ANOMALY DETECTOR NEW INSTANCES HIGH SCORED INSTANCES, POTENTIAL NOVEL CATEGORIES REJECT HIGH ANOMALY SCORES SIMILAR INSTANCES PREDICTION LABEL/RETRAIN MODEL ALERT WHEN CUMULATED ANOMALY SCORE DATA ANALYST
- 24. #MLSEV 24 Novel Categories Demo Steel plates faults dataset • Each instance represents a faulty steel plate with fault type label • Objective: predict fault type given a faulty steel plate … … … X_Min X_Max Y_Min Y_Max Pixels Areas X_Perim Y_Perim 42 50 270900 270944 267 17 44 645 651 2538079 2538108 108 10 30 829 835 1553913 1553931 71 8 19 853 860 369370 369415 176 13 45 1289 1306 498078 498335 2409 60 260 430 441 100250 100337 630 20 87 413 446 138468 138883 9052 230 432 190 200 210936 210956 132 11 20 330 343 429227 429253 264 15 26 74 90 779144 779308 1506 46 167 106 118 813452 813500 442 13 48 Orientation_Index Luminosity_Index SigmoidOfAreas Fault 0.8182 -0.2913 0.5822 Pastry 0.7931 -0.1756 0.2984 Bumps 0.6667 -0.1228 215 Bumps 0.8444 -0.1568 0.5212 Dirty 0.9338 -0.1992 1.0 Stains 0.8736 -0.2267 0.9874 Pastry 0.9205 0.2791 1.0 Stains 0.5 0.1841 0.3359 Bumps 0.5 -0.1197 0.5593 Pastry 0.9024 -0.0651 1.0 Pastry 0.75 -0.1093 0.8612 Pastry 28 fields total BigML Gallery
- 25. #MLSEV 25 Summary • Novel plates faults categories could be spotted with this method • Model degradation in general can be monitored with anomaly detectors