Semi-Supervised Learning In An Adversarial Environment
2 likes1,218 views
The document discusses the engineering challenges and methodologies in applying semi-supervised learning and clustering algorithms, particularly in the context of account security and fraud detection at Uber. It highlights the importance of feature selection, scalability issues, and the use of tools like DBSCAN and PCA for clustering in large datasets. The document outlines the goals for automating anomaly detection and enhancing system efficiency in real-time applications.
Semi-Supervised Learning In An Adversarial Environment
- 1. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent. Uber Risk Team Semi-supervised Learning In An Adversarial Environment Karthik Ramasamy Gaurav Agarwal
- 2. About us • Data Science Manager and founding member of - Account Security and Privacy • Previously: ‐ 2 years at my own startup ‐ 4.5 years at LinkedIn (founding member of the security team) • Interests: ‐ Applying ML, recently deep learning, in security and fraud problems ‐ Building scalable infra for ML systems Karthik Ramasamy
- 3. About us • Senior engineer and founding member of Account security team • 2 years at Uber working on fraud problems • Previously: ‐ 4 years at Microsoft working on NLP Question answering systems • Interests: ‐ Intersection of distributed systems and ML ‐ Natural language processing Gaurav Agarwal
- 4. Focus of the talk • DS algorithms and process • Features ‐ Not covered in this talk ‐ Arbitrary names will be used to describe features • Engineering architecture • Engineering challenges
- 6. Adversarial Problem Not-Hotdog Classifier Fake Account Classifier
- 7. Adversarial Problem Recall for a specific model % of null features
- 8. Account Takeovers (ATO) Single IP IPs in same class c/b Targeted Attacks on specific accounts Phishing and Malware Attacks Massive Botnet with >100K IPs Proxy IPs across world Easy to detect Hard to detect
- 11. Clustering Algorithm: DBSCAN 1. Method 1 - use labels to tune hyperparameter a. Solves the hyperparameter tuning issue 2. Method 2 - use labels as constraints in the clustering algorithm a. Challenges i. Working with week labels ii. Scalability Semi-Supervised Clustering Approach
- 12. Login Clusters 2016 *PCA used for visualization
- 13. Login Clusters 2017 *PCA used for visualization
- 14. ML Challenges • Feature Selection ‐ Manual feature selection ‐ Aggregate features are better ‐ Feature normalization is very important ▪ Features like trip fare and #trips are bad features ▪ % of UberX trips for a user is a good feature ▪ All features having same scale like % of X • Feature evolution in adversarial environment • Scalability ‐ DBSCAN for large dataset (10’s of millions) takes long time to fit • No online DBSCAN
- 15. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent. Engineering Architecture
- 16. Goals ● Minimum human input for unknown anomalies ● Automatic action support ● Low latency for sync scenarios eg: Login ● Easily reusable for other use cases eg: Bot signups ● Good ML library support
- 17. Query Flow Login Client Feature Gathering ML Models Actions Risk Gateway Rule Engine Other Stores Error code Challenge thrown Async call Parallel calls Sync callChallenges: ● 2FA ● Captcha ● ... Streaming Counters Clustering Features
- 19. Engineering Challenges ● Python vs Scala perf for streaming case ● DBScan limitations in Spark ● Window aggregations limited to 30 minutes ● JOINS with feedback signals in realtime
- 20. Production Setup ● Batch: 7 days worth of data, run DBSCAN hourly ● Streaming: 60 minutes moving window, run streaming k-means ● Used feedback signal success ratios to mark clusters as good, bad or unknown ● Bad clusters: Always throw ● Good clusters: Small % of attempts ● Unknown clusters: X% of attempts
- 21. Results Good Clusters Bad Clusters Unknown Clusters
- 22. GPU DBSCAN using Faiss from FB
- 23. Thank you!