Causal inference in online systems: Methods, pitfalls and best practices
Download as PPTX, PDF3 likes11,513 views
The document discusses the comparison of two algorithms (old and new) for predicting click-through rates (CTR) for recommendations on the Windows Store. It analyzes user behavior data and reports various metrics such as activity levels and stratified estimates, suggesting that the new algorithm (b) slightly increases overall CTR compared to the old algorithm (a). Additionally, it highlights that a significant portion of recommendation-driven visits may not be causal, indicating the complexity of accurately assessing algorithm impact.
![> nrow(user_app_visits_A)
[1] 1,000,000
> length(unique(user_app_visits_A$user_id))
[1] 10,000
> length(unique(user_app_visits_A$product_id))
[1] 990
> length(unique(user_app_visits_A$category))
[1] 10
82](https://image.slidesharecdn.com/causalinferencetutorial1-161022181021/85/Causal-inference-in-online-systems-Methods-pitfalls-and-best-practices-82-320.jpg)
![> user_app_visits_B = read.csv("user_app_visits_B.csv")
> naive_observational_estimate <- function(user_visits){
# Naive observational estimate
# Simply the fraction of visits that resulted in a recommendation click-
through.
est =
summarise(user_visits,
naive_estimate=sum(is_rec_visit)/length(is_rec_visit))
return(est)
}
> naive_observational_estimate(user_app_visits_A)
naive_estimate
[1] 0.200768
> naive_observational_estimate(user_app_visits_B)
naive_estimate
[1] 0.226467
85](https://image.slidesharecdn.com/causalinferencetutorial1-161022181021/85/Causal-inference-in-online-systems-Methods-pitfalls-and-best-practices-85-320.jpg)
![> stratified_by_activity_estimate(user_app_visits_A)
Source: local data frame [4 x 2]
activity_level stratified_estimate
1 1 0.1248852
2 2 0.1750483
3 3 0.2266394
4 4 0.2763522
> stratified_by_activity_estimate(user_app_visits_B)
Source: local data frame [4 x 2]
activity_level stratified_estimate
1 1 0.1253469
2 2 0.1753933
3 3 0.2257211
4 4 0.2749867
87](https://image.slidesharecdn.com/causalinferencetutorial1-161022181021/85/Causal-inference-in-online-systems-Methods-pitfalls-and-best-practices-87-320.jpg)
![> stratified_by_category_estimate(user_app_visits_A)
Source: local data frame [10 x 2]
category stratified_estimate
1 1 0.1758294
2 2 0.2276829
3 3 0.2763157
4 4 0.1239860
5 5 0.1767163
… … …
> stratified_by_category_estimate(user_app_visits_B)
Source: local data frame [10 x 2]
category stratified_estimate
1 1 0.2002127
2 2 0.2517528
3 3 0.3021371
4 4 0.1503150
5 5 0.1999519
… … …
88](https://image.slidesharecdn.com/causalinferencetutorial1-161022181021/85/Causal-inference-in-online-systems-Methods-pitfalls-and-best-practices-88-320.jpg)
![> naive_observational_estimate(user_app_visits_A)
naive_estimate
[1] 0.200768
> ranking_discontinuity_estimate(user_app_visits_A)
discontinuity_estimate
[1] 0.121362
40% of app visits coming from recommendation click-
throughs are not causal.
Could have happened even without the
recommendation system.
93](https://image.slidesharecdn.com/causalinferencetutorial1-161022181021/85/Causal-inference-in-online-systems-Methods-pitfalls-and-best-practices-93-320.jpg)
Causal inference in online systems: Methods, pitfalls and best practices
- 2. 2
- 3. 3
- 4. 4
- 5. 5
- 6. Use these correlations to make a predictive model. Future Activity -> f(number of friends, logins in past month) 6
- 7. 7
- 8. 8
- 9. 9
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. 14
- 15. 15
- 16. 16
- 17. 17
- 18. 18
- 19. 19 Old Algorithm (A) New Algorithm (B) 50/1000 (5%) 54/1000 (5.4%)
- 20. 20 Old Algorithm (A) New Algorithm (B) 10/400 (2.5%) 4/200 (2%) Old Algorithm (A) New Algorithm (B) 40/600 (6.6%) 50/800 (6.2%) 0 2 4 6 8 Low-activity High-activity CTR
- 21. Is Algorithm A better? Old algorithm (A) New Algorithm (B) CTR for Low- Activity users 10/400 (2.5%) 4/200 (2%) CTR for High- Activity users 40/600 (6.6%) 50/800 (6.2%) Total CTR 50/1000 (5%) 54/1000 (5.4%) 21
- 22. 22
- 23. Average comment length decreases over time. 23 But for each yearly cohort of users, comment length increases over time.
- 24. 24
- 25. 25
- 26. 26
- 29. 29
- 30. 30
- 31. 31
- 32. 32
- 33. 33
- 34. 34
- 35. 35
- 36. 36
- 37. 37
- 38. 38
- 39. 39
- 40. 40
- 41. 41Dunning (2002), Rosenzweig-Wolpin (2000)
- 42. 42
- 43. 43
- 44. 44
- 45. 45
- 46. 46
- 47. 47
- 48. 48
- 49. 49
- 50. 50
- 51. 51
- 52. 52
- 53. 53
- 54. 54
- 55. 55 Does new Algorithm B increase CTR for recommendations on Windows Store, compared to old algorithm A?
- 56. Does new Algorithm B increase CTR for recommendations on Windows Store, compared to old algorithm A? 56
- 57. 57
- 58. 58
- 59. 59
- 60. 60
- 61. 61
- 62. 62
- 63. 63
- 64. 64
- 65. 65
- 66. 𝑷𝒓𝒐𝒑𝒆𝒏𝒔𝒊𝒕𝒚 𝑁𝑒𝑤𝐴𝑙𝑔𝑜 𝑈𝑠𝑒𝑟𝑖 = 𝑳𝒐𝒈𝒊𝒔𝒕𝒊𝒄(𝑎 𝑐𝑎𝑡1, 𝑎 𝑐𝑎𝑡2, … 𝑎 𝑐𝑎𝑡𝑛) Compare CTR between users with the same propensity score. 66
- 67. 67
- 68. 68
- 70. 70
- 71. 71
- 72. 72
- 74. 74
- 75. http://www.github.com/amit-sharma/causal-inference- tutorial amshar@microsoft.com 75
- 77. 77
- 78. 78
- 79. 79
- 80. 80
- 81. 81
- 82. > nrow(user_app_visits_A) [1] 1,000,000 > length(unique(user_app_visits_A$user_id)) [1] 10,000 > length(unique(user_app_visits_A$product_id)) [1] 990 > length(unique(user_app_visits_A$category)) [1] 10 82
- 83. 83
- 84. 84
- 85. > user_app_visits_B = read.csv("user_app_visits_B.csv") > naive_observational_estimate <- function(user_visits){ # Naive observational estimate # Simply the fraction of visits that resulted in a recommendation click- through. est = summarise(user_visits, naive_estimate=sum(is_rec_visit)/length(is_rec_visit)) return(est) } > naive_observational_estimate(user_app_visits_A) naive_estimate [1] 0.200768 > naive_observational_estimate(user_app_visits_B) naive_estimate [1] 0.226467 85
- 86. 86
- 87. > stratified_by_activity_estimate(user_app_visits_A) Source: local data frame [4 x 2] activity_level stratified_estimate 1 1 0.1248852 2 2 0.1750483 3 3 0.2266394 4 4 0.2763522 > stratified_by_activity_estimate(user_app_visits_B) Source: local data frame [4 x 2] activity_level stratified_estimate 1 1 0.1253469 2 2 0.1753933 3 3 0.2257211 4 4 0.2749867 87
- 88. > stratified_by_category_estimate(user_app_visits_A) Source: local data frame [10 x 2] category stratified_estimate 1 1 0.1758294 2 2 0.2276829 3 3 0.2763157 4 4 0.1239860 5 5 0.1767163 … … … > stratified_by_category_estimate(user_app_visits_B) Source: local data frame [10 x 2] category stratified_estimate 1 1 0.2002127 2 2 0.2517528 3 3 0.3021371 4 4 0.1503150 5 5 0.1999519 … … … 88
- 89. 89
- 90. 90
- 91. 91
- 92. 92
- 93. > naive_observational_estimate(user_app_visits_A) naive_estimate [1] 0.200768 > ranking_discontinuity_estimate(user_app_visits_A) discontinuity_estimate [1] 0.121362 40% of app visits coming from recommendation click- throughs are not causal. Could have happened even without the recommendation system. 93
- 94. 94