Paris ML meetup
4,590 likes98,274 views
Netflix uses machine learning and algorithms to power recommendations for over 69 million members across more than 50 countries. They experiment with a wide range of algorithms including regression, matrix factorization, deep neural networks, and more. Some lessons learned are to first build an offline experimentation framework with clear metrics, consider distribution from the start, and design production code to also support experimentation. The goal is to efficiently iterate experiments and smoothly implement successful models in production.
Paris ML meetup
- 2. Machine Learning @ Netflix (and some lessons learned) Yves Raimond (@moustaki) Research/Engineering Manager Search & Recommendations Algorithm Engineering
- 4. Netflix scale ● > 69M members ● > 50 countries ● > 1000 device types ● > 3B hours/month ● 36% of peak US downstream traffic
- 5. Recommendations @ Netflix ● Goal: Help members find content to watch and enjoy to maximize satisfaction and retention ● Over 80% of what people watch comes from our recommendations ● Top Picks, Because you Watched, Trending Now, Row Ordering, Evidence, Search, Search Recommendations, Personalized Genre Rows, ...
- 6. ▪ Regression (Linear, logistic, elastic net) ▪ SVD and other Matrix Factorizations ▪ Factorization Machines ▪ Restricted Boltzmann Machines ▪ Deep Neural Networks ▪ Markov Models and Graph Algorithms ▪ Clustering ▪ Latent Dirichlet Allocation ▪ Gradient Boosted Decision Trees/Random Forests ▪ Gaussian Processes ▪ … Models & Algorithms
- 8. Build the offline experimentation framework first
- 9. When tackling a new problem ● What offline metrics can we compute that capture what online improvements we’ re actually trying to achieve? ● How should the input data to that evaluation be constructed (train, validation, test)? ● How fast and easy is it to run a full cycle of offline experimentations? ○ Minimize time to first metric ● How replicable is the evaluation? How shareable are the results? ○ Provenance (see Dagobah) ○ Notebooks (see Jupyter, Zeppelin, Spark Notebook)
- 10. When tackling an old problem ● Same… ○ Were the metrics designed when first running experimentation in that space still appropriate now?
- 11. Think about distribution from the outermost layers
- 12. 1. For each combination of hyper-parameter (e.g. grid search, random search, gaussian processes…) 2. For each subset of the training data a. Multi-core learning (e.g. HogWild) b. Distributed learning (e.g. ADMM, distributed L-BFGS, …)
- 13. When to use distributed learning? ● The impact of communication overhead when building distributed ML algorithms is non-trivial ● Is your data big enough that the distribution offsets the communication overhead?
- 14. Example: Uncollapsed Gibbs sampler for LDA (more details here)
- 15. Design production code to be experimentation-friendly
- 16. Idea Data Offline Modeling (R, Python, MATLAB, …) Iterate Implement in production system (Java, C++, …) Missing post- processing logic Performance issues Actual outputProduction environment (A/B test) Code discrepancies Final model Data discrepancies Example development process
- 17. Avoid dual implementations Shared Engine Experiment code Production code ProductionExperiment
- 19. We’re hiring! Yves Raimond (@moustaki)