Moment-Based Estimation for Hierarchical Models in Apache Spark with Kyle Schmaus
0 likes675 views
The document discusses Kyle Schmaus's presentation on moment-based estimation for hierarchical models at Stitch Fix, emphasizing the integration of this technique within Apache Spark for improved computational efficiency. It contrasts traditional estimation methods with non-iterative moment-based methods, highlighting their benefits and limitations, particularly regarding statistical efficiency. Additionally, the implementation details and performance metrics of the models used in logistic analysis on large datasets are reviewed.
![Hierarchical Models
14#DSSAIS17
• ![#$] = ()* +$, + .$/$
• /$ ~ 1 0, Σ
• 5 ∈ 1, … , 9
• ,, /$, and Σ are unknown](https://image.slidesharecdn.com/7kyleschmaus-180613193746/85/Moment-Based-Estimation-for-Hierarchical-Models-in-Apache-Spark-with-Kyle-Schmaus-14-320.jpg)
Moment-Based Estimation for Hierarchical Models in Apache Spark with Kyle Schmaus
- 1. Kyle Schmaus, Stitch Fix Moment-Based Estimation for Hierarchical Models in Apache Spark #DSSAIS17
- 2. Overview • About Stitch Fix • Hierarchical (Mixed Effects) Models • Moment Based Estimation • Spark Implementation & Application 2#DSSAIS17
- 7. Stitch Fix 7#DSSAIS17 Machine learning Human curation Algorithmic recommendations
- 8. Stitch Fix • Algorithms Team: 80+ Data Scientists and Data Engineers • Data Integrates into every aspect of the business • Our Blog: multithreaded.stitchfix.com 8#DSSAIS17
- 11. Shared Model ℙ(sale) = "#$ %$ & ' 11#DSSAIS17 ℙ(sale) = "#$ %( & ' " ) = log ) 1 − )
- 12. Individual Model Per Group 12#DSSAIS17
- 13. Individual Model Per Group 13#DSSAIS17 ℙ(sale) = "#$ %&$ ℙ(sale) = "#$ %&' " ( = log ( 1 − (
- 14. Hierarchical Models 14#DSSAIS17 • ![#$] = ()* +$, + .$/$ • /$ ~ 1 0, Σ • 5 ∈ 1, … , 9 • ,, /$, and Σ are unknown
- 15. Simulation Results 15#DSSAIS17 • ! ∈ 1, … , & = 100 • )* = +*, + .*/* + 0* • dim , = dim /* = 11 × 1 • /* ~ 7 0, Σ • 0* ~ 7 0, 9 • Σ ~ :;< < << 9, 11 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1.00 1.25 1.50 1.75 3 4 5 6 7 log(n_obs) rmse model_type ● ● ● individual model mixed model shared model RMSE By N Observations Per Group
- 16. Software Implementations 16#DSSAIS17 lme4, nlme, mbest, … statsmodels MixedModels
- 17. Software Implementations 17#DSSAIS17 lme4, nlme, mbest, … statsmodels MixedModels
- 18. Likelihood Based Methods Expectation-Maximization, Variational Approximations, or Likelihood Maximization require an ! "#$ initial cost, then a series of iterations costing ! %#& , where • " is the number of total observations • # the number of fixed and random effects • % the number of groups 18#DSSAIS17
- 19. Moment Based Methods Using a moment-based approach laid out in Perry (2015) and implemented in the mbest package, we can achieve a non-iterative fit in ! "#$ + ! &#' steps. This can be trivially spread across ( processors. 19#DSSAIS17
- 20. Moment Based Methods Using a moment-based approach laid out in Perry (2015) and implemented in the mbest package, we can achieve a non-iterative fit in ! "#$ + ! &#' steps. This can be trivially spread across ( processors. This improvement in computational efficiency is paid for by sacrificing some statistical efficiency. 20#DSSAIS17
- 21. Moment Based Setup !" = $"% + '"(" + )" for * ∈ {1, … , 0} (" ∼ 3(0, Σ) )" ∼ 3(0, 89) (" and )" independent 21#DSSAIS17
- 22. Moment Based Setup • Define !" ≡ $" %" and &" ' ≡ (' )" ' • *" = $"( + %")" + -" ⟹ • *" = !"&" + -" • Estimate /&" = !" ' !" 0 !" ' 1", for each 3 ∈ 1, … , 7 • Note when !" is rank deficient, /&" is not an unbiased estimator 22#DSSAIS17
- 23. Moment Based Setup • !" = $"%"&" ' • (" = $"%"&") ' • *" = $"%"&"+ ' • ,ϕ = ./+ = ) 012 ∑"4) 5 6" − ,!" ̂9" + • : ≡ ∑"4) 5 <" 23#DSSAIS17
- 24. Estimate ! Ω ≡ $ %&' ( )%'*%)%' + ,!- = Ω/' $ %&' ( )%'*%)% + 01% is an unbiased estimator for !. 24#DSSAIS17
- 25. Estiamte ! Say we knew *, not just an estimate. Define • 6 ≡ ∑9:; < =9>?9(=9 A BC9 − =9; A *) VG H BC9 − =9; A * A ?9=9> A • I ≡ ∑9:; < =9>?9J9 K> ?9=9> A • Ω> ≡ ∑9:; < =9>?9=9> A ⨂=9>?9=9> A • vec{ QΣS} = Ω> K; vec{6 − VI} • QΣS is an unbiased estimator for Σ 25#DSSAIS17
- 26. Estiamte ! • "Σ$ is an unbiased estimator for Σ if you know 6 a priori. We don’t … • Instead, we use >?. This mean "Σ$ is not an unbiased estimator in practice. • It can be shown the bias is often negligible. • We can project "Σ$ to be Positive Semidefinite 26#DSSAIS17
- 27. Estimate !"| $%" • Using a gaussian approximation for p($67|87) and p(87), compute posterior distribution – p(87|$67) ∝ p($67|87) p(87) • There exists a formula for C7(Σ) such that – EF(87 G = C7I7J (I7 K $67 − I7M K N) – OVar(87 G = OQ7C7 27#DSSAIS17
- 28. That was a lot of math! Read Fast moment-based estimation for hierarchical models on arXiv, it’s a great paper 28#DSSAIS17
- 29. Moment Based Estimation Summary 29#DSSAIS17 • Estimate !"#
- 30. Moment Based Estimation Summary 30#DSSAIS17 • Estimate !"# • Use !"# to estimate $%
- 31. Moment Based Estimation Summary 31#DSSAIS17 • Estimate !"# • Use !"# to estimate $% • Use !"# to estimate &'(
- 32. Moment Based Estimation Summary 32#DSSAIS17 • Estimate !"# • Use !"# to estimate $% • Use !"# to estimate &'( • Use !"#, $%, &'( to estimate &Σ(
- 33. Moment Based Estimation Summary 33#DSSAIS17 • Estimate !"# • Use !"# to estimate $% • Use !"# to estimate &'( • Use !"#, $%, &'( to estimate &Σ( • Use !"#, $%, &'(, &Σ( to estimate +#|!"#
- 34. lme4 vs mbest 34#DSSAIS17 ● ● ● ● ● ● ● ● ● ● ● ● 1.02 1.04 1.06 1.08 1.10 3 4 5 6 log(n_obs) rmse model_type ● ● lmer mhglm RMSE By N Observations Per Group ● ● ● ● ● ● ● ● ● ● ● ● 0 2 4 6 3 4 5 6 log(n_obs) log(seconds) model_type ● ● lmer mhglm Seconds To Train By N Observations Per Group
- 35. Memory Constrained For Large N, M • Current software is memory constrained on a single machine • My macbook pro (2015) starts to run out of memory for – M = 100,000 – N = 10,000,000 35#DSSAIS17
- 38. Application Logistic Model with • N = 120,000,000 • M = 4,000,000 • p = q = 7 • Executors = 8, 16gb memory each • Running time approximately 40 minutes 38#DSSAIS17