VSSML17 L3. Clusters and Anomaly Detection
1 like752 views
The document discusses clustering and anomaly detection in machine learning, focusing on their definitions, methodologies, and practical applications. It explains clustering as an unsupervised learning technique used to find self-similar groups and identifies various clustering use cases such as customer segmentation and item discovery. Anomaly detection is also presented as a means to identify unusual instances or behaviors, useful for applications like fraud detection and data cleaning.
VSSML17 L3. Clusters and Anomaly Detection
- 1. Valencian Summer School in Machine Learning 3rd edition September 14-15, 2017
- 2. BigML, Inc 2 Clusters Finding Similarities Poul Petersen CIO, BigML, Inc
- 3. BigML, Inc 3Clusters What is Clustering? ā¢ An unsupervised learning technique ā¢ No labels necessary ā¢ Useful for ļ¬nding similar instances ā¢ Smart sampling/labelling ā¢ Finds āself-similar" groups of instances ā¢ Customer: groups with similar behavior ā¢ Medical: patients with similar diagnostic measurements ā¢ Deļ¬nes each group by a ācentroidā ā¢ Geometric center of the group ā¢ Represents the āaverageā member ā¢ Number of centroids (k) can be speciļ¬ed or determined
- 4. BigML, Inc 4Clusters Cluster Centroids date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51
- 5. BigML, Inc 5Clusters Cluster Centroids date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51 auth = pin amount ~ $100 Same: date: Mon != Wed customer: Sally != Bob account: 6788 != 3421 class: clothes != gas zip: 26339 != 46140 Different: date = Wed (2 out of 3) customer = Bob account = 3421 auth = pin class = gas zip = 46140 amount = $104 Centroid: similar
- 6. BigML, Inc 6Clusters Use Cases ā¢ Customer segmentation ā¢ Which customers are similar? ā¢ How many natural groups are there? ā¢ Item discovery ā¢ What other items are similar to this one? ā¢ Similarity ā¢ What other instances share a speciļ¬c property? ā¢ Recommender (almost) ā¢ If you like this item, what other items might you like? ā¢ Active learning ā¢ Labelling unlabelled data efļ¬ciently
- 7. BigML, Inc 7Clusters Customer Segmentation GOAL: Cluster the users by usage statistics. Identify clusters with a higher percentage of high LTV users. Since they have similar usage patterns, the remaining users in these clusters may be good candidates for up-sell. ā¢ Dataset of mobile game users. ā¢ Data for each user consists of usage statistics and a LTV based on in- game purchases ā¢ Assumption: Usage correlates to LTV 0% 3% 1%
- 8. BigML, Inc 8Clusters Similarity GOAL: Cluster the loans by application profile to rank loan quality by percentage of trouble loans in population ā¢ Dataset of Lending Club Loans ā¢ Mark any loan that is currently or has even been late as ātroubleā 0% 3% 7% 1%
- 9. BigML, Inc 9Clusters Active Learning GOAL: Rather than sample randomly, use clustering to group patients by similarity and then test a sample from each cluster to label the data. ā¢ Dataset of diagnostic measurements of 768 patients. ā¢ Want to test each patient for diabetes and label the dataset to build a model but the test is expensive*.
- 10. BigML, Inc 10Clusters Active Learning *For a more realistic example of high cost, imagine a dataset with a billion transactions, each one needing to be labelled as fraud/not- fraud. Or a million images which need to be labeled as cat/not-cat. 2323
- 11. BigML, Inc 11Clusters Item Discovery GOAL: Cluster the whiskies by flavor profile to discover whiskies that have similar taste. ā¢ Dataset of 86 whiskies ā¢ Each whiskey scored on a scale from 0 to 4 for each of 12 possible ļ¬avor characteristics. Smoky Fruity
- 12. BigML, Inc 12Clusters Clusters Demo #1
- 13. BigML, Inc 13Clusters Human Expert Cluster into 3 groupsā¦
- 14. BigML, Inc 14Clusters Human Expert
- 15. BigML, Inc 15Clusters Human Expert ā¢ Jesa used prior knowledge to select possible features that separated the objects. ā¢ āroundā, āskinnyā, āedgesā, āhardā, etc ā¢ Items were then clustered based on the chosen features ā¢ Separation quality was then tested to ensure: ā¢ met criteria of K=3 ā¢ groups were sufļ¬ciently ādistantā ā¢ no crossover
- 16. BigML, Inc 16Clusters Human Expert ā¢ Length/Width ā¢ greater than 1 => āskinnyā ā¢ equal to 1 => āroundā ā¢ less than 1 => invert ā¢ Number of Surfaces ā¢ distinct surfaces require āedgesā which have corners ā¢ easier to count Create features that capture these object differences
- 17. BigML, Inc 17Clusters Clustering Features Object Length / Width Num Surfaces penny 1 3 dime 1 3 knob 1 4 eraser 2.75 6 box 1 6 block 1.6 6 screw 8 3 battery 5 3 key 4.25 3 bead 1 2
- 18. BigML, Inc 18Clusters Plot by Features Num Surfaces Length / Width box block eraser knob penny dime bead key battery screw K-Means Key Insight: We can ļ¬nd clusters using distances in n-dimensional feature space K=3
- 19. BigML, Inc 19Clusters Plot by Features Num Surfaces Length / Width box block eraser knob penny dime bead key battery screw K-Means Find ābestā (minimum distance) circles that include all points
- 20. BigML, Inc 20Clusters K-Means Algorithm K=3
- 21. BigML, Inc 21Clusters K-Means Algorithm K=3 Repeat until centroids stop moving
- 22. BigML, Inc 22Clusters Features Matter Metal Other Wood
- 23. BigML, Inc 23Clusters Convergence Convergence guaranteed but not necessarily unique Starting points important (K++)
- 24. BigML, Inc 24Clusters Starting Points ā¢ Random points or instances in n-dimensional space ā¢ Chose points āfarthestā away from each other ā¢ but this is sensitive to outliers ā¢ k++ ā¢ the ļ¬rst center is chosen randomly from instances ā¢ each subsequent center is chosen from the remaining instances with probability proportional to its squared distance from the point's closest existing cluster center
- 25. BigML, Inc 25Clusters Scaling Matters price number of bedrooms d = 160,000 d = 1
- 26. BigML, Inc 26Clusters Other Tricks ā¢ What is the distance to a āmissing valueā? ā¢ What is the distance between categorical values? ā¢ What is the distance between text features? ā¢ Does it have to be Euclidean distance? ā¢ Unknown āKā?
- 27. BigML, Inc 27Clusters Distance to Missing? ā¢ Nonsense! Try replacing missing values with: ā¢ Maximum ā¢ Mean ā¢ Median ā¢ Minimum ā¢ Zero ā¢ Ignore instances with missing values
- 28. BigML, Inc 28Clusters Distance to Categorical? ā¢ Deļ¬ne special distance function: For two instances š„ and š¦ Ā and the categorical ļ¬eld š: ā¢ if š„ š Ā ļ¼ Ā š¦ š thenāØ (š„,š¦)distanceļ¼0 Ā (or ļ¬eld scaling value) āØ else āØ (š„,š¦)distanceļ¼1 Approach: similar to āk-prototypesā
- 29. BigML, Inc 29Clusters Distance to Categorical? animal favorite toy toy color cat ball red cat ball green d=0 d=0 d=1 cat laser red dog squeaky red d=1 d=1 d=0 D = 1 Then compute Euclidean distance between vectors D = ā2 Note: the centroid is assigned the most common category of the member instances
- 30. BigML, Inc 30Clusters Text Vectors 1 Cosine Similarity 0 -1 "hippo" "safari" "zebra" ā¦. 1 0 1 ā¦ 1 1 0 ā¦ 0 1 1 ā¦ Text Field #1 Text Field #2 Features(thousands) ā¢ Cosine Ā Similarity Ā ā¢ cos() between two vectors ā¢ 1 if collinear, 0 if orthogonal ā¢ only positive vectors: 0 Ā ā¤ Ā CS Ā ā¤ Ā 1 ā¢ Cosine Ā Distanceļ¼1ļ¼Cosine Ā Similarity Ā ā¢ CD(TF1, Ā TF2) Ā = Ā 0.5
- 31. BigML, Inc 31Clusters Finding K: G-Means
- 32. BigML, Inc 32Clusters Finding K: G-Means
- 33. BigML, Inc 33Clusters Finding K: G-Means Let K=2 Keep 1, Split 1 New K=3
- 34. BigML, Inc 34Clusters Finding K: G-Means Let K=3 Keep 1, Split 2 New K=5
- 35. BigML, Inc 35Clusters Finding K: G-Means Let K=5 K=5
- 36. BigML, Inc 36Clusters Clusters Demo #2
- 37. BigML, Inc 37Clusters Summary ā¢ Cluster Purpose ā¢ Unsupervised technique for ļ¬nding self-similar groups of instances ā¢ Number of centroids (k) can be inputed or computed ā¢ Outputs list of centroids ā¢ Conļ¬guration: ā¢ Algorithm: K-means / G-means ā¢ Cluster Parameter: k or critical value ā¢ Default missing / Summary ļ¬elds / Scales / Weights ā¢ Model Clusters ā¢ Centroid / Batchcentroids
- 38. BigML, Inc 2 Anomaly Detection Finding the Unusual Poul Petersen CIO, BigML, Inc
- 39. BigML, Inc 3Anomaly Detection What is Anomaly Detection? ā¢ An unsupervised learning technique ā¢ No labels necessary ā¢ Useful for ļ¬nding unusual instances ā¢ Filtering, ļ¬nding mistakes, 1-class classiļ¬ers ā¢ Finds instances that do not match ā¢ Customer: big or small spender for proļ¬le ā¢ Medical: healthy patient despite indicative diagnostics ā¢ Deļ¬nes each unusual instance by an āanomaly scoreā ā¢ in BigML: 0ļ¼normal, Ā 1ļ¼unusual, and 0.7 Ā ā« Ā 0.6 Ā ļ¹„0.5 ā¢ Standard deviation, distributions, etc
- 40. BigML, Inc 4Anomaly Detection Clusters date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51
- 41. BigML, Inc 5Anomaly Detection Clusters date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51 similar
- 42. BigML, Inc 6Anomaly Detection Anomaly Detection date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51
- 43. BigML, Inc 7Anomaly Detection Anomaly Detection date customer account auth class zip amount Mon Bob 3421 pin clothes 46140 135 Tue Bob 3421 sign food 46140 401 Tue Alice 2456 pin food 12222 234 Wed Sally 6788 pin gas 26339 94 Wed Bob 3421 pin tech 21350 2459 Wed Bob 3421 pin gas 46140 83 Thr Sally 6788 sign food 26339 51 anomaly ā¢ Amount $2,459 is higher than all other transactions ā¢ It is the only transaction ā¢ In zip 21350 ā¢ for the purchase class "tech"
- 44. BigML, Inc 8Anomaly Detection Use Cases ā¢ Unusual instance discovery - "exploration" ā¢ Intrusion Detection - "looking for unusual usage patterns" ā¢ Fraud - "looking for unusual behavior" ā¢ Identify Incorrect Data - "looking for mistakes" ā¢ Remove Outliers - "improve model quality" ā¢ Model Competence / Input Data Drift
- 45. BigML, Inc 9Anomaly Detection Removing Outliers ā¢ Models need to generalize ā¢ Outliers negatively impact generalization GOAL: Use anomaly detector to identify most anomalous points and then remove them before modeling. DATASET FILTERED DATASET ANOMALY DETECTOR CLEAN MODEL
- 46. BigML, Inc 10Anomaly Detection Diabetes Anomalies DIABETES SOURCE DIABETES DATASET TRAIN SET TEST SET ALL MODEL CLEAN DATASET FILTER ALL MODEL ALL EVALUATION CLEAN EVALUATION COMPARE EVALUATIONS ANAOMALY DETECTOR
- 47. BigML, Inc 11Anomaly Detection Anomaly Demo #1
- 48. BigML, Inc 12Anomaly Detection Intrusion Detection GOAL: Identify unusual command line behavior per user and across all users that might indicate an intrusion. ā¢ Dataset of command line history for users ā¢ Data for each user consists of commands, ļ¬ags, working directories, etc. ā¢ Assumption: Users typically issue the same ļ¬ag patterns and work in certain directories Per User Per Dir All User All Dir
- 49. BigML, Inc 13Anomaly Detection Fraud ā¢ Dataset of credit card transactions ā¢ Additional user proļ¬le information GOAL: Cluster users by profile and use multiple anomaly scores to detect transactions that are anomalous on multiple levels. Card Level User Level Similar User Level
- 50. BigML, Inc 14Anomaly Detection Model Competence ā¢ After putting a model it into production, data that is being predicted can become statistically different than the training data. ā¢ Train an anomaly detector at the same time as the model. GOAL: For every prediction, compute an anomaly score. If the anomaly score is high, then the model may not be competent and should not be trusted. Prediction T T Conļ¬dence 86% 84% Anomaly Score 0.5367 0.7124 Competent? Y N At Prediction TimeAt Training Time DATASET MODEL ANOMALY DETECTOR
- 51. BigML, Inc 15Anomaly Detection Univariate Approach ā¢ Single variable: heights, test scores, etc ā¢ Assume the value is distributed ānormallyā ā¢ Compute standard deviation ā¢ a measure of how āspread outā the numbers are ā¢ the square root of the variance (The average of the squared differences from the Mean.) ā¢ Depending on the number of instances, choose a āmultipleā of standard deviations to indicate an anomaly. A multiple of 3 for 1000 instances removes ~ 3 outliers.
- 52. BigML, Inc 16Anomaly Detection Univariate Approach measurement frequency outliersoutliers ā¢ Available in BigML API
- 53. BigML, Inc 17Anomaly Detection Benfordās Law ā¢ In real-life numeric sets the small digits occur disproportionately often as leading signiļ¬cant digits. ā¢ Applications include: ā¢ accounting records ā¢ electricity bills ā¢ street addresses ā¢ stock prices ā¢ population numbers ā¢ death rates ā¢ lengths of rivers ā¢ Available in BigML API
- 54. BigML, Inc 18Anomaly Detection Multivariate Matters
- 55. BigML, Inc 19Anomaly Detection Multivariate Matters
- 56. BigML, Inc 20Anomaly Detection Human Expert Most Unusual?
- 57. BigML, Inc 21Anomaly Detection Human Expert āRoundāāSkinnyā āCornersā āSkinnyā but not āsmoothā No āCornersā Not āRoundā Key Insight The āmost unusualā object is diļ¬erent in some way from every partition of the features. Most unusual
- 58. BigML, Inc 22Anomaly Detection Human Expert ā¢ Human used prior knowledge to select possible features that separated the objects. ā¢ āroundā, āskinnyā, āsmoothā, ācornersā ā¢ Items were then separated based on the chosen features ā¢ Each cluster was then examined to see which object ļ¬t the least well in its cluster and did not ļ¬t any other cluster
- 59. BigML, Inc 23Anomaly Detection Human Expert ā¢ Length/Width ā¢ greater than 1 => āskinnyā ā¢ equal to 1 => āroundā ā¢ less than 1 => invert ā¢ Number of Surfaces ā¢ distinct surfaces require āedgesā which have corners ā¢ easier to count ā¢ Smooth - true or false Create features that capture these object differences
- 60. BigML, Inc 24Anomaly Detection Anomaly Features Object Length / Width Num Surfaces Smooth penny 1 3 TRUE dime 1 3 TRUE knob 1 4 TRUE eraser 2.75 6 TRUE box 1 6 TRUE block 1.6 6 TRUE screw 8 3 FALSE battery 5 3 TRUE key 4.25 3 FALSE bead 1 2 TRUE
- 61. BigML, Inc 25Anomaly Detection length/width > 5 smooth? box blockeraser knob penny/dime bead key battery screw num surfaces = 6 length/width =1 length/width < 2 Know that āsplitsā matter - donāt know the order TrueFalse TrueFalse TrueFalse FalseTrue TrueFalse Random Splits
- 62. BigML, Inc 26Anomaly Detection Isolation Forest Grow a random decision tree until each instance is in its own leaf āeasyā to isolate āhardā to isolate Depth Now repeat the process several times and use average Depth to compute anomaly score: 0 (similar) -> 1 (dissimilar)
- 63. BigML, Inc 27Anomaly Detection Isolation Forest Scoring D = 3 D = 6 D = 2 S=0.45 Map avg depth to ļ¬nal score f1 f2 f3 i1 red cat ball i2 red cat ball i3 red cat box i4 blue dog pen For the instance, i2 Find the depth in each tree
- 64. BigML, Inc 28Anomaly Detection Model Competence ā¢ A low anomaly score means the loan is similar to the modeled loans. ā¢ A high anomaly score means you should not trust the model. Prediction T T Conļ¬dence 86% 84% Anomaly Score 0.5367 0.7124 Competent? Y N OPEN LOANS PREDICTION ANOMALY SCORE CLOSED LOAN MODEL CLOSED LOAN ANOMALY DETECTOR
- 65. BigML, Inc 29Anomaly Detection Anomaly Demo #2
- 66. BigML, Inc 30Anomaly Detection 1-Class Classiļ¬er? ā¢ You place an advertisement in a local newspaper ā¢ You collect demographic information about all responders ā¢ Now you want to market in a new locality with direct letters ā¢ To optimize mailing costs, need to predict who will respond ā¢ But, can not distinguish not interested from didnāt see the ad ā¢ Train an anomaly detector on the 1-class data ā¢ Pick the households with the lowest scores for mailing: ā¢ If a household has a low anomaly score, then they are āsimilarā to enough of your positive responders and therefore may respond as well ā¢ If an individual has a high anomaly score, then they are dissimilar from all previous responders and therefore are less likely to respond.
- 67. BigML, Inc 31Anomaly Detection Summary ā¢ Anomaly detection is the process of ļ¬nding unusual instances ā¢ Some techniques and how they work: ā¢ Univariate: standard deviation ā¢ Benfordās law ā¢ Isolation Forest ā¢ Applications ā¢ Filtering to improve models ā¢ Finding mistakes, fraud, and intruders ā¢ Knowing when to retrain a model (competence) ā¢ 1-class classiļ¬ers ā¢ In generalā¦ unsupervised learning techniques: ā¢ Require more ļ¬nesse and interpretation ā¢ Are more commonly part of a multistep workļ¬ow