Concept Drift: Monitoring Model Quality In Streaming ML Applications
4 likes3,915 views
The document discusses various challenges and solutions in model quality, focusing on issues such as concept drift, sensor failure, and population change. It presents techniques for monitoring and adapting machine learning models using methods like statistical process control and clustering, as well as evaluating their efficiency and pros/cons. Additionally, it highlights explicit and implicit mechanisms for adaptation within nonstationary environments, suggesting different methods based on the type of data drift encountered.
![statistical process control
• Drift Detection Method [DDM]
• # of errors is Binomial:
• alert:](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-29-320.jpg)
![statistical process control
• Early Drift Detection Method [EDDM]
• distance between errors
better for gradual drift
• warn & start caching:
• alert and reset max:
• Drift Detection Method [DDM]
• # of errors is Binomial:
• alert:](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-30-320.jpg)
![sequential analysis
• Linear Four Rates [LFR]
• stationary data => constant contingency table
0 1
0 TN FN
1 FP TP
Predicted
True](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-32-320.jpg)
![sequential analysis
• Linear Four Rates [LFR]
• stationary data => constant contingency table
• calculate four rates
0 1
0 TN FN
1 FP TP
Predicted
True](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-33-320.jpg)
![sequential analysis
• Linear Four Rates [LFR]
• stationary data => constant contingency table
• calculate four rates
• incremental updates
0 1
0 TN FN
1 FP TP
Predicted
True](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-34-320.jpg)
![sequential analysis
• Linear Four Rates [LFR]
• stationary data => constant contingency table
• calculate four rates
• incremental updates
• test for change
• Monte Carlo sampling
for significance level
• Bonferoni correction
for correlated tests
• O(1)
• Better than (E)DDM
for class imbalance
0 1
0 TN FN
1 FP TP
Predicted
True](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-35-320.jpg)
![error distribution monitoring
• ADaptive WINdowing [ADWIN]
• Consider all partitions of a window
• Drop the last element if any
• Efficient version O(log W)
• Data structure for windows ~ exponential histograms
• Drop last window rather than last element
w0 w1
prediction
errors](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-37-320.jpg)
![feature distribution monitoring
• Monitor individual features
• Many ways to compare:
• Pearson correlation [Change of Concept - CoC]
• Hellinger distance [HDDDM] ~ O(DB)
• PCA to reduce the number of features to track (top [PCA-1] or bottom [PCA-2] n%)
w0
w1
color
size
samples](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-43-320.jpg)
![model-dependent monitoring
• Not all changes matter
• Posterior probability estimate
• Use [A-distance] ~ generalized Kolmogorov-Simirnov distance
• designed to be less sensitive to irrelevant changes
L1-distance KS-distance A-distance](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-45-320.jpg)
![model-dependent monitoring
• [Margin] distribution
• rank statistic on density estimates for a
binary representation of the data,
• compare average margins of a linear classifier
induced by the 1-norm SVM
• based on the average zero-one or sigmoid error
rate of an SVM classifier
• Generalized margin [MD3]:
• Embed base classifier in a
Random Feature Bagged Ensemble
• Margin == high disagreement region of the ensemble
m
argin
“margin”
“margin”](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-46-320.jpg)
![implicit mechanisms for adaptation
• Online NonStationary boosting [ONSboost]
• NonStationary Random Forests [NSRF]
• Dynamic Weighted Majority [DWM]
• Learn++ for NonStationary Environments [Learn++.NSE]
retire / decay
recurring
train new member
ensemble 1 ensemble (N-1) ensemble N
Ensemble Based Adaptation](https://image.slidesharecdn.com/oevmlmonitoring20180322final-180326120725/85/Concept-Drift-Monitoring-Model-Quality-In-Streaming-ML-Applications-56-320.jpg)
Concept Drift: Monitoring Model Quality In Streaming ML Applications
- 4. * same data generating distribution (Some algorithms tolerate violation of this to a certain degree.) training set operation=* core problem
- 11. classifiermargin can active learning help? color size
- 12. classifiermargin can active learning help? color size
- 13. classifiermargin can active learning help? color size
- 14. classifiermargin can active learning help? color size
- 15. classifiermargin can active learning help? color size
- 16. classifiermargin can active learning help? color size
- 17. a better solution data classifier feature extraction predictions
- 18. a better solution data classifier feature extraction predictions monitoring
- 19. a better solution data classifier feature extraction predictions monitoring
- 20. a better solution data classifier feature extraction predictions monitoring
- 21. a better solution data classifier feature extraction predictions monitoring
- 22. a better solution data classifier labeling - feature extraction predictions monitoring
- 23. a better solution data classifier labeling -change detection adaptation feature extraction predictions
- 24. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 25. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 26. which method?
- 27. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 28. ML theory: samples errors
- 29. statistical process control • Drift Detection Method [DDM] • # of errors is Binomial: • alert:
- 30. statistical process control • Early Drift Detection Method [EDDM] • distance between errors better for gradual drift • warn & start caching: • alert and reset max: • Drift Detection Method [DDM] • # of errors is Binomial: • alert:
- 31. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 32. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table 0 1 0 TN FN 1 FP TP Predicted True
- 33. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates 0 1 0 TN FN 1 FP TP Predicted True
- 34. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates • incremental updates 0 1 0 TN FN 1 FP TP Predicted True
- 35. sequential analysis • Linear Four Rates [LFR] • stationary data => constant contingency table • calculate four rates • incremental updates • test for change • Monte Carlo sampling for significance level • Bonferoni correction for correlated tests • O(1) • Better than (E)DDM for class imbalance 0 1 0 TN FN 1 FP TP Predicted True
- 36. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 37. error distribution monitoring • ADaptive WINdowing [ADWIN] • Consider all partitions of a window • Drop the last element if any • Efficient version O(log W) • Data structure for windows ~ exponential histograms • Drop last window rather than last element w0 w1 prediction errors
- 38. resampling • Prediction loss over random permutations vs. ordered training data • Parallel permutation test version available • Still expensive • Only method directly applicable to regression setting • Side note: Even with finite training set, drift could be problematic if model is developed naively.
- 39. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 40. clustering / novelty detection • OLINDDA: K-means, periodically merge unknown to known or flag • MINAS: micro-clusters, incremental stream clustering • DETECTNOD: Discrete Cosine Transform to estimate distances efficiently • Woo-ensemble: Treat outliers as potential emerging class centroids • ECSMiner: Store and use cluster summary efficiently • GC3: Grid based clustering size color
- 41. Curse of Dimensionality clustering / novelty detection size color • OLINDDA: K-means, periodically merge unknown to known or flag • MINAS: micro-clusters, incremental stream clustering • DETECTNOD: Discrete Cosine Transform to estimate distances efficiently • Woo-ensemble: Treat outliers as potential emerging class centroids • ECSMiner: Store and use cluster summary efficiently • GC3: Grid based clustering
- 42. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 43. feature distribution monitoring • Monitor individual features • Many ways to compare: • Pearson correlation [Change of Concept - CoC] • Hellinger distance [HDDDM] ~ O(DB) • PCA to reduce the number of features to track (top [PCA-1] or bottom [PCA-2] n%) w0 w1 color size samples
- 44. monitor how? supervised unsupervised statistical process control sequential analysis error distribution monitoring clustering / novelty detection feature distribution monitoring model-dependent monitoring
- 45. model-dependent monitoring • Not all changes matter • Posterior probability estimate • Use [A-distance] ~ generalized Kolmogorov-Simirnov distance • designed to be less sensitive to irrelevant changes L1-distance KS-distance A-distance
- 46. model-dependent monitoring • [Margin] distribution • rank statistic on density estimates for a binary representation of the data, • compare average margins of a linear classifier induced by the 1-norm SVM • based on the average zero-one or sigmoid error rate of an SVM classifier • Generalized margin [MD3]: • Embed base classifier in a Random Feature Bagged Ensemble • Margin == high disagreement region of the ensemble m argin “margin” “margin”
- 47. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 48. explicit mechanisms for adaptation W stationary W drift ADWIN Drop the last sub-window if threshold is exceeded. = Adaptively shrink window during drift.
- 49. explicit mechanisms for adaptation * Adaptation goes through a similar refinement process. JIT w 0 m 0 m 1 m 2 m 3 m 4 I 0 I 1 I 2 I 3 I 4 change detected * w 1 w 2 w 3 w 4
- 50. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 51. explicit mechanisms for adaptation Biased Reservoir Sampling bias: capacity: overwrite / exchange randomly w/ Prob{ %full } or append
- 52. adapt how? explicit mechanisms implicit mechanisms windowing weighting sampling pure methods ensemble methods
- 53. implicit mechanisms for adaptation Ensemble Based Adaptation ensemble 1 ensemble (N-1) ensemble N train new member
- 54. implicit mechanisms for adaptation retire / decay train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 55. implicit mechanisms for adaptation retire / decay recurring train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 56. implicit mechanisms for adaptation • Online NonStationary boosting [ONSboost] • NonStationary Random Forests [NSRF] • Dynamic Weighted Majority [DWM] • Learn++ for NonStationary Environments [Learn++.NSE] retire / decay recurring train new member ensemble 1 ensemble (N-1) ensemble N Ensemble Based Adaptation
- 57. which method? Method Efficiency Pros Cons Notes DDM/EDDM O(1) no data stored label cost false alarms sampling necessary in case of fast data, microservices architecture ideal LFR O(1) class imbalance OK label cost ADWIN O(log W) better change localization label cost JIT O(log W) no labels required only for abrupt changes best localization
- 58. which method? Method Efficiency Pros Cons Notes ECSMiner / GC3 O(W 2 / k) O(G log C) emerging concepts clusterable drift only use if emerging concepts expected HDDDM O(DB) no labels not for population drift or class imbalance better when combined with PCA A-distance O(log W) no labels less false positives compared to HDDDM good choice for unsupervised Margin / MD3 Learning, detection, adaptation bundled reduced false alarms must use feature bagged ensembles best choice but must commit to using the specific machine learning algorithmsEnsemble methods recurring concepts large batches
- 60. Check out these resources: Dean’s book Webinars etc.Fast Data Architectures for Streaming Applications Getting Answers Now from Data Sets that Never End By Dean Wampler, Ph. D., VP of Fast Data Engineering 60 LIGHTBEND.COM/LEARN
- 61. Serving Machine Learning Models A Guide to Architecture, Stream Processing Engines, and Frameworks By Boris Lublinsky, Fast Data Platform Architect 61 LIGHTBEND.COM/LEARN