The Factorization Machines algorithm for building recommendation system - Paweł Łagodziński, SAS Institute
1 like1,963 views
The document discusses the application of factorization machines in building recommender systems, highlighting their effectiveness in sparse data scenarios. It outlines the challenges of traditional regression and matrix factorization models while showcasing the advantages of factorization machines, such as improved performance in various data science challenges. The final sections mention specific implementations, including a shot recommendation system for NBA coaches and provide insights on when to use factorization machines.
![FEBRUARY 22, 2018, WARSAW
Nero
JuliusCaesar
Cleopatra
SleeplessinSeatle
PrettyWomen
Casablanca
1 0 0 0 0 0 0
2 0 0 0 0 0 0
3 0 0 0 0 0 0
4 0 0 -1 0 0 0
5 0 0 -1 0 0 0
6 0 0 1 0 0 0
7 0 0 -1 0 0 0
≈
History
Romance
1 1 0
2 1 0
3 1 0
4 1 1
5 -1 1
6 -1 1
7 -1 1
Nero
JuliusCaesar
Cleopatra
SleeplessinSeatle
PrettyWomen
Casablanca
History 1 1 1 0 0 0
Romance 0 0 1 1 1 1
Rating: -1- dislike, 0 – neutral, 1 - like
History
Romance
Both x
Laten factor models - matrix factorization
• correlations -> lower rank matrix
• strong correlations -> works for sparse data
after C.C. Aggarwal [1]
Nero
JuliusCaesar
Cleopatra
SleeplessinSeatle
PrettyWomen
Casablanca
1 1 1 1 0 0 0
2 1 1 1 0 0 0
3 1 1 1 0 0 0
4 1 1 1 1 1 1
5 -1 -1 -1 1 1 1
6 -1 -1 1 1 1 1
7 -1 -1 -1 1 1 1
• Intuition behind matrix factorization with low (2) rank matrix
𝑊 𝐻 𝑇𝑌 𝑌 − 𝑌](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-7-320.jpg)
![FEBRUARY 22, 2018, WARSAW
Factorization Machines
• Concept generalizing regression and factorization models
• Introduced in 2010 (S. Randel [2])
• Very good performance in data science challenges (after S. Randal, MLConf, 2014):
• Click prediction - KDDCup 2012: Track 2 (3rd place), Criteon Display Advertising Challenge (1st place)
• Rating prediction – Netlix prize, Movielens, KDDCup 2011, EMI Music Data Science Hackathon (2nd
place)
• Social link prediction (social media) - KDDCup 2012: Track 1 (2nd place)
• Recommend given name - ECML/PKDD Discovery Challenge 2013 (1st place on-line track, 2nd place
off-line track)
• Predicting result of next exam question - KDDCup 2010, Grockit Challenge (1st place)
• Prediction of auction sale price –Kaggle, Blue Book for Bulldozers (1st place)](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-10-320.jpg)
![FEBRUARY 22, 2018, WARSAW
FactMac – Factorization Machines in SAS Viya
• 2-nd degree Factorization Machines
• Learning with „HOGWILD!”
• Hyper-parameters optimization with Grid Search, Genetic Algorithm
• Different deployment options
• SAS Viya
• In-database
• In-stream (SAS ESP)
• Considered for the next releases
• Field-Aware Factorization Machines, FFM [3] – better predictions than FM
• Warm-up restart – faster retraining
• Other loss functions
2
3
1
1
2
3](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-17-320.jpg)
![FEBRUARY 22, 2018, WARSAW
Shot Recommender System for NBA Coaches
• Application of Factorization Machines in SAS Viya to sport data
• Presented during 2016 KDD Large-Scale Sports Analytics [5]
• Goal – recommend shot types for players
• Data - recorded shots taken during the 2015-2016 NBA games
• shots for 420 players
• for shot: player, action type, shot zone range, shot zone area, result,
and other characteristics.
• Model for shot prediction (FM 2-d, k=25, p = 404), log-odds as a target
?
Image source: Wikipedia](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-18-320.jpg)
![FEBRUARY 22, 2018, WARSAW
Where to use Factorization Machines?
• Rule of Thumb
• design matrix is very sparse (mostly missing data, 90% missing or more)
• the cardinality of the nominal variables is very high and each level of the nominal variables
has only a few representatives in the data
• you have good reason to believe the design matrix has low rank
• Findings in [3], seems to confirm above. FFM was working better for CTR prediction but no so good for more
dense data (web phishing, census income)
• Our (local SAS office) experiments, still in progress, showed that FM even for not very sparse data
can produce robust set of baseline predictions. More specialized and tailored models can
outperform FM but requires more effort in development
• FactMac was tested for SAS Customer Intelligence 360 solution with good results.](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-23-320.jpg)
![FEBRUARY 22, 2018, WARSAW
References
• [1] Charu C. Aggrwal. „Recommender Systems, The Textbook”. Springer. 2016.
• [2] Steffen Rendle. “Factorization Machines.” In Proceedings of the 10th IEEE
International Conference on Data Mining (ICDM). Piscataway, NJ: Institute of Electrical
And Electronics Engineers. 2010.
• [3] Yuchin Juan et al., „Field-aware Factorization Machines for CTR Prediction”. RecSys
'16 Proceedings of the 10th ACM Conference on Recommender Systems. 2016.
• [4] Jason Lee et al., „Practical Large-Scale Optimization for Max-Norm Regularization”,
Advances in Neural Information Processing Systems 23 (NIPS). 2010.
• [5] Reymond E. Wrigt et al. „Shot Recommender System for NBA Coaches”. KDD. 2016](https://image.slidesharecdn.com/pawellagodzinski-180327085708/85/The-Factorization-Machines-algorithm-for-building-recommendation-system-Pawel-Lagodzinski-SAS-Institute-25-320.jpg)
The Factorization Machines algorithm for building recommendation system - Paweł Łagodziński, SAS Institute
- 1. FEBRUARY 22, 2018, WARSAW Factorization Machines for building Recommender System Paweł Łagodziński, SAS Institute Poland
- 2. FEBRUARY 22, 2018, WARSAW Recommender Systems • Primary goal – recommend most relevant items to users Plenty of success stories: • music, movies • retail products • ad verbs • news • query results • social links Users Items Context Recommender System
- 3. FEBRUARY 22, 2018, WARSAW Recommender Systems • Key challenges • sparse data • extremely large data • cold start • Other issues • different ratings types • difficult performance evolution
- 4. FEBRUARY 22, 2018, WARSAW Regression models • Linear regression • Polynomial regression (2-d) – input vector with p predictors y – predicted rating – number of parameters O( ҧ𝑝) – learning complexity 1+p – observed rating 1+p+p2 – number of parameters O( ҧ𝑝2 ) – learning complexity
- 5. FEBRUARY 22, 2018, WARSAW Regression models • Pros • Combines real-value and categorical features • Useful for regression and classification problems • Fast learning for linear model • Cons • Linear regression does not include user-item interactions • Polynomial regression model includes, but only in theory for sparse data • For sparse data pairwise interactions are very rarely observed, in consequences parameters cannot be estimated directly from the data
- 6. FEBRUARY 22, 2018, WARSAW Laten factor models - matrix factorization 𝑌 ∈ ℝ|𝑈|×|𝐼| matrix of observed ratings i1 i2 i3 i4 u1 2 3 1 ? u2 ? 5 ? 1 u3 ? ? 4 1 u4 2 3 ? 2 I - items U-users • If matrix Y is low-rank it can be approximated with 𝑌 𝑌 ≈ 𝑌 where k is the rank of matrix 𝒀 (the size of latent factors vector)𝑌 = 𝑊𝐻 𝑇 , W ∈ ℝ 𝑈 ×𝑘, H ∈ ℝ|𝐼|×𝑘 Rating predictions ො𝑦 𝑢𝑖 = 𝑗=1 𝑘 𝑤 𝑢𝑗 ∙ ℎ𝑗𝑖 =< 𝒘 𝑢 𝒉𝑖 >
- 7. FEBRUARY 22, 2018, WARSAW Nero JuliusCaesar Cleopatra SleeplessinSeatle PrettyWomen Casablanca 1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0 4 0 0 -1 0 0 0 5 0 0 -1 0 0 0 6 0 0 1 0 0 0 7 0 0 -1 0 0 0 ≈ History Romance 1 1 0 2 1 0 3 1 0 4 1 1 5 -1 1 6 -1 1 7 -1 1 Nero JuliusCaesar Cleopatra SleeplessinSeatle PrettyWomen Casablanca History 1 1 1 0 0 0 Romance 0 0 1 1 1 1 Rating: -1- dislike, 0 – neutral, 1 - like History Romance Both x Laten factor models - matrix factorization • correlations -> lower rank matrix • strong correlations -> works for sparse data after C.C. Aggarwal [1] Nero JuliusCaesar Cleopatra SleeplessinSeatle PrettyWomen Casablanca 1 1 1 1 0 0 0 2 1 1 1 0 0 0 3 1 1 1 0 0 0 4 1 1 1 1 1 1 5 -1 -1 -1 1 1 1 6 -1 -1 1 1 1 1 7 -1 -1 -1 1 1 1 • Intuition behind matrix factorization with low (2) rank matrix 𝑊 𝐻 𝑇𝑌 𝑌 − 𝑌
- 8. FEBRUARY 22, 2018, WARSAW Matrix factorization based models • Many specific variants of factorization models were proposed • Examples: • MF • Time Senstive Factorization Models • User-item matrix factorization • SVD++ • Factorized kNN • timeSVD • timeTF • Tensore Factorization Models • Sequential Models • PARAFAC • PITF • FMC • FPMC (after S. Randal, MLConf, 2014)
- 9. FEBRUARY 22, 2018, WARSAW Matrix factorization based models • Pros • Enable modelling interactions (user-item-…) even if such events are not observed in data • Effective on sparse data • Cons • Learning algorithms are tailored to each specific model • In result, it is hard to adjust model to the problem under consideration
- 10. FEBRUARY 22, 2018, WARSAW Factorization Machines • Concept generalizing regression and factorization models • Introduced in 2010 (S. Randel [2]) • Very good performance in data science challenges (after S. Randal, MLConf, 2014): • Click prediction - KDDCup 2012: Track 2 (3rd place), Criteon Display Advertising Challenge (1st place) • Rating prediction – Netlix prize, Movielens, KDDCup 2011, EMI Music Data Science Hackathon (2nd place) • Social link prediction (social media) - KDDCup 2012: Track 1 (2nd place) • Recommend given name - ECML/PKDD Discovery Challenge 2013 (1st place on-line track, 2nd place off-line track) • Predicting result of next exam question - KDDCup 2010, Grockit Challenge (1st place) • Prediction of auction sale price –Kaggle, Blue Book for Bulldozers (1st place)
- 11. FEBRUARY 22, 2018, WARSAW Factorization Machines - model • Factorization Machines (2nd degree) – input vector with p predictors y – observed rating – predicted rating 1+p+kp – number of parameters O( ҧ𝑝𝑘) – learning complexity (lower than ҧ𝑝2 for 2-d polynomial regression) • FM of higher degrees possible but not used in practice • Parameters can be estimated with use of universal optimization methods, Stochastic Gradient Decent for instance
- 12. FEBRUARY 22, 2018, WARSAW Factorization Machines • Features engineering enables great expressiveness of FM model • Matrix / tensor data can be easily transformed into features vector Input Y i1 i2 i3 i4 u1 2 3 1 ? u2 ? 5 ? 1 u3 ? ? 4 1 User(U) Item (I) U I y 1 u1 i1 2 2 u1 i2 3 3 u1 i3 1 4 u2 i2 5 5 u2 i4 1 6 u3 i3 4 7 u3 i4 1 target y x1 1 0 0 1 0 0 0 2 y1 x2 1 0 0 0 1 0 0 3 y2 x3 1 0 0 0 0 1 0 1 y3 x4 0 1 0 0 1 0 0 5 y4 x5 0 1 0 0 0 0 1 1 y5 x6 0 0 1 0 0 1 0 4 y6 x7 0 0 1 0 0 0 1 1 y7 u1 u2 u3 i1 i2 i3 i4 Users Items x Features vector (one-hot encoding)
- 13. FEBRUARY 22, 2018, WARSAW Factorization Machines • Applying FM model target y x1 1 0 0 1 0 0 0 2 y1 x2 1 0 0 0 1 0 0 3 y2 x3 1 0 0 0 0 1 0 1 y3 x4 0 1 0 0 1 0 0 5 y4 x5 0 1 0 0 0 0 1 1 y5 x6 0 0 1 0 0 1 0 4 y6 x7 0 0 1 0 0 0 1 1 y7 u1 u2 u3 i1 i2 i3 i4 Users Items x Features vector FM is equal to combination of MF (interaction user-item) and biases for user and item
- 14. FEBRUARY 22, 2018, WARSAW Factorization Machines • Adding context categorical information gives other MF model FM is equivalent to PITF model, one of extension of MF model ො𝑦 𝑢𝑖𝑐 = 𝑤0 + 𝑤 𝑢 + 𝑤𝑖 + 𝑤𝑐 +< 𝒗 𝑢 𝒗𝑖 >+< 𝒗 𝑢 𝒗 𝑐 >+< 𝒗𝑖 𝒗 𝑐 > target y x1 1 0 0 1 0 0 0 0 0 1 2 y1 x2 1 0 0 0 1 0 0 1 0 0 3 y2 x3 1 0 0 0 0 1 0 1 0 0 1 y3 x4 0 1 0 0 1 0 0 0 1 0 5 y4 x5 0 1 0 0 0 0 1 1 0 0 1 y5 x6 0 0 1 0 0 1 0 0 1 0 4 y6 x7 0 0 1 0 0 0 1 0 0 1 1 y7 u1 u2 u3 i1 i2 i3 i4 c1 c2 c3 Context Features vector x Users Items
- 15. FEBRUARY 22, 2018, WARSAW Factorization Machines • Factorization Machines offers combination of regression and factorization models • Low rank matrix approximation enables estimation of unobserved interactions • Effective for sparse and extremely sparse data • Flexible through features engineering • One of the most disruptive algorithms of recent years
- 16. FEBRUARY 22, 2018, WARSAW FactMac – Factorization Machines in SAS Viya
- 17. FEBRUARY 22, 2018, WARSAW FactMac – Factorization Machines in SAS Viya • 2-nd degree Factorization Machines • Learning with „HOGWILD!” • Hyper-parameters optimization with Grid Search, Genetic Algorithm • Different deployment options • SAS Viya • In-database • In-stream (SAS ESP) • Considered for the next releases • Field-Aware Factorization Machines, FFM [3] – better predictions than FM • Warm-up restart – faster retraining • Other loss functions 2 3 1 1 2 3
- 18. FEBRUARY 22, 2018, WARSAW Shot Recommender System for NBA Coaches • Application of Factorization Machines in SAS Viya to sport data • Presented during 2016 KDD Large-Scale Sports Analytics [5] • Goal – recommend shot types for players • Data - recorded shots taken during the 2015-2016 NBA games • shots for 420 players • for shot: player, action type, shot zone range, shot zone area, result, and other characteristics. • Model for shot prediction (FM 2-d, k=25, p = 404), log-odds as a target ? Image source: Wikipedia
- 19. FEBRUARY 22, 2018, WARSAW Shot Recommender System for NBA Coaches • Biases can be very informative
- 20. FEBRUARY 22, 2018, WARSAW Shot Recommender System for NBA Coaches • Latent factors analysis
- 21. FEBRUARY 22, 2018, WARSAW Shot Recommender System for NBA Coaches • Shot recommendations (plogit):
- 22. FEBRUARY 22, 2018, WARSAW Shot Recommender System for NBA Coaches • Demonstration of the concept on NCAA* basketball data • The National Collegiate Athletic Association (NCAA) is a non-profit organization which regulates athletes of 1,281 institutions, conferences, and individuals (source: wikipedia). • Recent news for Kaggle’rs, new competation launched 2 days ago
- 23. FEBRUARY 22, 2018, WARSAW Where to use Factorization Machines? • Rule of Thumb • design matrix is very sparse (mostly missing data, 90% missing or more) • the cardinality of the nominal variables is very high and each level of the nominal variables has only a few representatives in the data • you have good reason to believe the design matrix has low rank • Findings in [3], seems to confirm above. FFM was working better for CTR prediction but no so good for more dense data (web phishing, census income) • Our (local SAS office) experiments, still in progress, showed that FM even for not very sparse data can produce robust set of baseline predictions. More specialized and tailored models can outperform FM but requires more effort in development • FactMac was tested for SAS Customer Intelligence 360 solution with good results.
- 24. FEBRUARY 22, 2018, WARSAW Thank You !
- 25. FEBRUARY 22, 2018, WARSAW References • [1] Charu C. Aggrwal. „Recommender Systems, The Textbook”. Springer. 2016. • [2] Steffen Rendle. “Factorization Machines.” In Proceedings of the 10th IEEE International Conference on Data Mining (ICDM). Piscataway, NJ: Institute of Electrical And Electronics Engineers. 2010. • [3] Yuchin Juan et al., „Field-aware Factorization Machines for CTR Prediction”. RecSys '16 Proceedings of the 10th ACM Conference on Recommender Systems. 2016. • [4] Jason Lee et al., „Practical Large-Scale Optimization for Max-Norm Regularization”, Advances in Neural Information Processing Systems 23 (NIPS). 2010. • [5] Reymond E. Wrigt et al. „Shot Recommender System for NBA Coaches”. KDD. 2016
- 26. FEBRUARY 22, 2018, WARSAW