Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments
0 likes368 views
This document summarizes a technique for transforming ratio metrics into linear metrics to improve the efficiency of online controlled experiments. The technique was developed by researchers at Yandex, Russia's largest internet company. It introduces a linearization transformation that preserves the directionality and statistical significance of ratio metrics, allowing ratio metrics like average session length to be analyzed like user-level metrics. The researchers prove theorems showing the transformation maintains directionality and asymptotic normality of significance tests. They applied the technique to 390 A/B tests at Yandex, finding it improved sensitivity of results by 34% compared to existing approaches like the bootstrap test or delta method.
![µA(X)=avgu in AX(u)
µB(X)=avgu in BX(u)
Overall Evaluation Criterion
(OEC) for the group B
Overall Evaluation Criterion
(OEC) for the group A
Calculate the OEC for each
group as the mean value
Δ(x) VS 0
Δ(X) = µB(X) – µA(X)
the evaluated update is
positive or negative
Statistical
significance test
the difference is caused by
a noise or
the treatment effect
(e.g., Student’s t-test)
Overall Evaluation
Criterion (OEC)
[Kohavi et al., DMKD’2009]
Overall Acceptance
Criterion (OAC)
[Drutsa et al., CIKM’2015]
Sensitivity
Directionality](https://image.slidesharecdn.com/ocestalk2-181218173111/85/Consistent-Transformation-of-Ratio-Metrics-for-Efficient-Online-Controlled-Experiments-8-320.jpg)
Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments
- 2. Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments Roman Budylin, Alexey Drutsa, Ilya Katsev, Valeriya Tsoy
- 3. About Yandex • The largest internet company in Europe • ”Google of Russia”, “Amazon of Russia”, “Uber of Russia” and so on • More than 50M DAU
- 6. Group A Group B Split them randomly trafficofusers Variant for A Variant for B Expose to one of two variants of the service e.g., the current production version e.g., an evaluated update
- 7. Group A Group B Split them randomly trafficofusers Variant for A Variant for B Expose to one of two variants of the service e.g., the current production version e.g., an evaluated update Calculate a key measure for each user X(uA1) … Calculate the OEC for each group as the mean value e.g., X(u) is the number of sessions of the user u X(uA2) X(uA3) X(uA4) X(uA5) X(uB1) … X(uB2) X(uB3) X(uB4) X(uB5) µA(X)=avgu in AX(u) µB(X)=avgu in BX(u) Overall Evaluation Criterion (OEC) for the group B Overall Evaluation Criterion (OEC) for the group A
- 8. µA(X)=avgu in AX(u) µB(X)=avgu in BX(u) Overall Evaluation Criterion (OEC) for the group B Overall Evaluation Criterion (OEC) for the group A Calculate the OEC for each group as the mean value Δ(x) VS 0 Δ(X) = µB(X) – µA(X) the evaluated update is positive or negative Statistical significance test the difference is caused by a noise or the treatment effect (e.g., Student’s t-test) Overall Evaluation Criterion (OEC) [Kohavi et al., DMKD’2009] Overall Acceptance Criterion (OAC) [Drutsa et al., CIKM’2015] Sensitivity Directionality
- 9. OEC levels 1. User level metrics 2. Non-user level metric For example ratio OEC:
- 10. Example – average length of session
- 11. Example – average length of session
- 12. Example – average length of session > 30m
- 13. Example – average length of session
- 14. Example – average length of session Let user u has sessions with lengths Then the average of lengths of all sessions is equal to
- 15. Problem For ratio OEC t-test sensitivity improvement techniques
- 16. Problem For ratio OEC t-test sensitivity improvement techniques
- 17. Problem For ratio OEC t-test sensitivity improvement techniques
- 18. Existing approaches • Bootstrap test Problem: computational expensiveness • Delta method Problem: does not allow to apply directly regression adjustment techniques
- 20. Our paper, WSDM 2018 21 Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments Roman Budylin, Alexey Drutsa, Ilya Katsev, Valeriya Tsoy Data Quality metrics • OEC metrics • Guard rail metrics • Local feature/Diagnosticmetrics
- 21. Our paper, WSDM 2018 22 Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments Roman Budylin, Alexey Drutsa, Ilya Katsev, Valeriya Tsoy The best comment: «This is too good to be true!» (someone from Facebook) Data Quality metrics • OEC metrics • Guard rail metrics • Local feature/Diagnosticmetrics
- 22. We found a transformation such that Our contribution
- 23. We found a transformation such that Ratio OEC User level OEC Our contribution
- 24. We found a transformation such that Ratio OEC User level OEC NB: Preserve directionality and significance level! Our contribution
- 25. Let we have a ratio OEC: Consider the next expression: And let us use its average as a metric: Now we got a linearization OEC: Linearization
- 26. Let A and B be the control and the experiment. Let us denote Theorem 1: Let be positive. Then for any the next is true: Theorems: directionality
- 27. Theorem 2: Let be positive and . Let be the t-statistic applied for the OEC and let be the be the asymt. standard normal statistic of obtained via the Delta method. If then then under the null hypothesis that 1. the t-statistics is asymptotically normal 2. converges to 1 by probability Theorems: significance level
- 29. Experiments • Internet search, 2013-2016, 390 experiments • The transformation + regression adjustment • +34% of sensitivity
- 30. Experiments
- 31. The code