Intro to Factorization Machines
12 likes4,964 views
The document discusses a presentation on factorization machines at a data science meetup, highlighting influential machine learning papers suggested by Peter Norvig, including Rendle's work. It outlines the use of factorization machines in handling high-dimensional sparse data and describes their capability to model interactions through various examples. Additionally, it mentions hyperparameters and regularization to avoid overfitting in the model training process.
Intro to Factorization Machines
- 1. Factorization Machines St. Petersburg Data Science Meetup, May, 29th, 2015 @facultyofwonder Rutarget/Segmento
- 3. Q: What, say, 3 recent papers in machine learning do you think will be influential to directing the cutting edge of research these days? Peter Norvig: I’ve never been able to pick lasting papers in the past, so don’t trust me now, but here are a few: ● Rendle’s “Factorization Machines” ● Wang et al. “Bayesian optimization in high dimensions via random embeddings” ● Dean et al. “Fast, Accurate Detection of 100,000 Object Classes on a Single Machine” http://blog.teamleada.com/2014/08/ask-peter-norvig/
- 5. Data Lots of categorical features Sparse settings Pairwise interactions Hashing trick?
- 6. Linear model
- 8. Factorization breaking the independence of interaction parameters
- 9. Example U = {Alice (A), Bob (B), Charlie (C), . . .} I = {Titanic (TI), Notting Hill (NH), Star Wars (SW), Star Trek (ST), . . .}
- 10. Example {(A, TI, 2010-1, 5),(A, NH, 2010-2, 3),(A, SW, 2010-4, 1), (B, SW, 2009-5, 4),(B, ST, 2009-8, 5), (C, TI, 2009-9, 1),(C, SW, 2009-12, 5)} Interaction between Alice and Star Trek to predict rating? Zero interaction? B-SW and C-SW are similar A and C are different ST and SW are similar A-SW and A-ST are to be similar
- 11. Complexity Number of parameters: 1 + p + k * p Linear to the input size and the size of factorization
- 12. Regularization Many parameters, prone to overfitting L2 regularization
- 13. LR+FM+SGD
- 14. Learning
- 16. Hyperparameters Number of factors Regularization Learning rate Initial weights
- 17. FFM:ideas Features can be grouped into fields: users, movies, context, SSPs, publishers, whatever Better use this information Factor vector per field
- 18. Summary Factorized interactions High sparsity is OK for parameters estimation