RecSysOps: Best Practices for Operating a Large-Scale Recommender System
- 1. RecSysOps: Best Practices for Operating a Large-Scale Recommender System Ehsan (Mohammad) Saberian, Justin Basilico RecSys 2021 2021-09-27 @ehsan_saberian, @JustinBasilico, @NetflixResearch
- 2. Large scale RecSys is a complex operation
- 3. RecSys environment is dynamic It changes every second!
- 4. New members
- 10. How to ensure that our RecSys is working correctly?
- 11. RecSysOps: Lessons we learned while operating a large RecSys
- 12. Benefits: Reduce firefighting time Focus on innovation Build trust with our stakeholders
- 18. Detect issues quickly The most challenging part There are endless potential issues Some of them we don’t know yet!
- 19. Detection lesson 1: Implement all the known best practices Unit test, Integration test MLOps: data/metrics check CICD, regular retraining
- 20. Detection lesson 1: Implement all the known best practices Unit test, Integration test MLOps: data/metrics check CICD, regular retraining
- 21. Detection lesson 2: Monitor end-to-end your own way
- 22. Detection lesson 2: Monitor end-to-end your own way Don’t rely only on partner teams’ audits What does correct data look like from your perspective?
- 23. Detection lesson 3: Understand your stakeholders’ concerns Stakeholders: members and items
- 24. Detection lesson 3.1: Every time a member plays something that is ranked low by the model; it is a potential issue
- 25. Detection lesson 3.1: Every time a member plays something that is ranked low by the model; it is a potential issue Monitor and analyze them Get inspiration for future innovations
- 26. Detection lesson 3.2: Engage with teams responsible for items and understand their concerns
- 27. Detection lesson 3.2: Engage with teams responsible for items and understand their concerns Is an item cold-started properly? Is production bias hurting an item?
- 28. Detection lesson 3.2: Engage with teams responsible for items and understand their concerns Build tools to detect their concerns and integrated them in your system
- 30. Can you predict issues before they happen?
- 31. Netflix case: Is it possible to predict if an item is going to cold-start properly 7 days before its launch date?
- 32. Yes, we can train a model to predict production model’s behaviour on day of launch Flag any item with unexpected prediction and investigate
- 33. Prediction Lesson: Try to predict issues before they happen
- 35. Step 1: Reproduce issue in isolation
- 36. Step 1: Reproduce issue in isolation Need sufficient advanced logging
- 37. Step 2: Input data issue or model issue?
- 38. Step 2.1: Input data issue? Are input values right?
- 39. Step 2.1: Input data issue? Are input values right? Trick: use similar items or members to estimate range of typical values
- 40. Step 2.1: Input data issue? Are input values right? Example: language of an item was set up incorrectly
- 41. Step 2.2: Model issue? Need to inspect/interpret model
- 42. Step 2.2: Model issue? Need to inspect/interpret model There are many tools, SHAP, LIME ..
- 43. Step 2.2: Model issue? Need to inspect/interpret model Example: missing values were handled incorrectly
- 44. Diagnosis lessons: Set up logging to reproduce issue Develop tools to check validity of inputs Develop tools to inspect models
- 46. It’s like software engineering Hotfixes and long term solutions
- 47. Hotfixes in ML are challenging! Models are highly optimized and hotfix modification will lead to suboptimality
- 48. Resolution lesson 1: Have a collection of hotfixes ready Understand their costs and trade-offs
- 49. Resolution lesson 2: With every issue, make RecsysOps better
- 51. Final lesson: Make RecSysOps frictionless
- 52. Run checks on a regular basis If human judgment is needed, make all required information ready Be able to deploy hotfixes with couple of clicks