Iterating over statistical models: NCAA tournament edition
2 likes10,307 views
The document discusses statistical modeling in the context of the NCAA March Madness basketball tournament, emphasizing the importance of treating statistical models as code for better collaboration and analysis. It highlights different statistical models used to predict outcomes of games, such as logistic regression and the Bradley-Terry model, and details the steps involved in Bayesian data analysis. The document also provides historical context on basketball and the evolution of its rules.
Iterating over statistical models: NCAA tournament edition
- 1. ITERATING OVER STATISTICAL MODELS NCAA TOURNAMENT EDITION
- 2. Daniel Lee WHY ARE YOU LISTENING TO ME? ▸ Stan developer http://mc-stan.org ▸ Researcher at Columbia ▸ Co-founder of Stan Group training / statistical support / consulting ▸ email: bearlee@alum.mit.edu twitter: @djsyclik / @mcmc_stan web: http://syclik.com
- 4. WHAT IS MARCH MADNESS?
- 5. WHAT IS MARCH MADNESS?
- 6. CHAMPIONSHIP GAME: VILLANOVA VS NORTH CAROLINA
- 7. P(VILLANOVA WIN)? CHAMPIONSHIP GAME: VILLANOVA VS NORTH CAROLINA
- 8. P(VILLANOVA WIN)? VILLANOVA WILDCATS VS. NORTH CAROLINA TAR HEELS ▸ 1:00. 70 - 69 ▸ 0:35. 72 - 69 ▸ 0:23. 72 - 71
- 9. P(VILLANOVA WIN)? VILLANOVA WILDCATS VS. NORTH CAROLINA TAR HEELS ▸ 1:00. 70 - 69 ▸ 0:35. 72 - 69 ▸ 0:23. 72 - 71 ▸ 0:13. 74 - 71
- 10. P(VILLANOVA WIN)? VILLANOVA WILDCATS VS. NORTH CAROLINA TAR HEELS ▸ 1:00. 70 - 69 ▸ 0:35. 72 - 69 ▸ 0:23. 72 - 71 ▸ 0:13. 74 - 71 ▸ 0:06. 74 - 74
- 11. P(VILLANOVA WIN)? VILLANOVA WILDCATS VS. NORTH CAROLINA TAR HEELS ▸ 1:00. 70 - 69 ▸ 0:35. 72 - 69 ▸ 0:23. 72 - 71 ▸ 0:13. 74 - 71 ▸ 0:06. 74 - 74 ▸ 0:00. 77 - 74. Villanova wins.
- 14. WRITING CODE IS A DISCIPLINE
- 15. WRITING CODE IS A DISCIPLINE ▸ design patterns ▸ testing ▸ code review ▸ maintenance ▸ modularity ▸ collaboration CAN BE
- 16. IS STATISTICAL MODELING A DISCIPLINE? ▸ Art or science? ▸ Models have names ▸ Statistical model vs implementation ▸ Collaboration on the statistical model?
- 17. TREAT STATISTICAL MODELS AS CODE WHAT DO WE NEED TO DO ▸ Elevate statistical models: first class entity ▸ Modularize ▸ Language ▸ Discuss subtle details ▸ Collaborate
- 18. TREAT STATISTICAL MODELS AS CODE STAN GETS US CLOSE ▸ statistical modeling language ▸ domain-specific language has its own grammar; not R or BUGS! ▸ rstan shinystan, RStudio integration, rstanarm ▸ open-source core libraries are new BSD ▸ Stan program ▸ plain text ▸ plays nicely with source repositories ▸ imperative language Note: Stan isn’t the only thing you can do this with
- 20. MARCH MADNESS BASKETBALL HISTORY ▸ 1891 ▸ Dr. James Naismith. Springfield, MA ▸ Non-contact conditioning ▸ 13 simple rules ▸ Peach basket
- 21. MARCH MADNESS BASKETBALL HISTORY ▸ 1891 ▸ Dr. James Naismith. Springfield, MA ▸ Non-contact conditioning ▸ 13 simple rules ▸ Peach basket 10. The umpire shall be judge of the men and shall note the fouls and notify the referee when three consecutive fouls have been made. He shall have power to disqualify men according to Rule 5.
- 22. MARCH MADNESS BASKETBALL NOW ▸ 2 x 20 min half ▸ Increasing score ▸ Points increment by 2, 3, and 1 ▸ 5 players, unlimited substitutions ▸ player DQ: 5th foul ▸ Bonus: 7 team fouls Double bonus: 10 team fouls
- 23. MARCH MADNESS DATA ▸ 351 NCAA Division 1 Men’s basketball teams ▸ 33 conferences ▸ 5421 games ▸ 24 - 35 games per team ▸ Max 3 observations
- 25. IS THIS BIG DATA?
- 26. TALL DATA VS WIDE DATA ▸ Tall data lots of replications ▸ Wide data lots of fields day, home, score, ot, fgm, fga, 3pm, 3pa, 3a, fta, ftm, or, dr, ast, to, stl, blk, pf
- 27. THREE STEPS OF BAYESIAN DATA ANALYSIS
- 28. ANDREW GELMAN IN BDA THE THREE STEPS OF BAYESIAN DATA ANALYSIS 1. Set up full probability model 2. Condition on observed data 3. Evaluate the fit of the model
- 29. ANDREW GELMAN IN BDA THE THREE STEPS OF BAYESIAN DATA ANALYSIS 1. Set up full probability model Write a Stan program 2. Condition on observed data 3. Evaluate the fit of the model
- 30. ANDREW GELMAN IN BDA THE THREE STEPS OF BAYESIAN DATA ANALYSIS 1. Set up full probability model Write a Stan program 2. Condition on observed data Run RStan 3. Evaluate the fit of the model
- 31. ANDREW GELMAN IN BDA THE THREE STEPS OF BAYESIAN DATA ANALYSIS 1. Set up full probability model Write a Stan program 2. Condition on observed data Run RStan 3. Evaluate the fit of the model R, ShinyStan, posterior predictive checks
- 33. ITERATING OVER STATISTICAL MODELS STATISTICAL MODEL #1 ▸ Only 2015-2016 matters ▸ Teams have a latent ability ▸ “logistic regression” ▸ “Bradley-Terry model” y ⇠ bernoulli(logit 1 (✓1 ✓2))
- 35. P(VILLANOVA > UNC | DATA) = 0.73
- 38. ITERATING OVER STATISTICAL MODELS TREAT STATISTICAL MODELS AS CODE ▸ Statistical model in a separate file ▸ Git ▸ Testing ▸ Inspection of fit ▸ Backtest on historical data ▸ Priors ▸ “Model #1”
- 39. IN WRITING, YOU MUST KILL ALL YOUR DARLINGS. Willliam Faulkner
- 40. ITERATING OVER STATISTICAL MODELS STATISTICAL MODEL #2 ▸ Home court advantage! ▸ Teams have a latent ability ▸ “logistic regression” ▸ “Bradley-Terry model” y ⇠ bernoulli(logit 1 (↵ + ✓1 ✓2))
- 42. P(VILLANOVA > UNC | DATA) = 0.52
- 44. ITERATING OVER STATISTICAL MODELS STATISTICAL MODEL #3 ▸ Assumptions: ▸ Only 2015-2016 matters ▸ Teams have a latent ability ▸ Model points ▸ Add a home court advantage
- 48. ITERATING OVER STATISTICAL MODELS HOW DID WE DO? ▸ Kaggle: 87 / 608 ▸ What went wrong? ▸ What’s next?
- 50. THANKS ▸ Collaborative effort ▸ NCAA modeling: Rob Trangucci ▸ Stan team Andrew Gelman, Bob Carpenter, Matt Hoffman, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, Allen Riddell, Marco Ignacio, Jeff Arnold, Mitzi Morris, Rob Goedman, Brian Lau, Jonah Gabry, Alp Kucukelbir, Robert Grant, Dustin Tran, Krzysztof Sakrejda, Aki Vehtari, Rayleigh Lei, Sebastian Weber HELP ▸ http://mc-stan.org ▸ stan-users mailing list ▸ Stan Group Inc. http://stan.fit training / statistical support / consulting ▸ bearlee@alum.mit.edu / @djsyclik / @mcmc_stan