Bootstrapping of PySpark Models for Factorial A/B Tests
1 like472 views
1. Factorial A/B testing involves running multiple experiments simultaneously by assigning each visitor to a variant in all tests, allowing for faster results than isolated tests. 2. Bootstrapping can be used to estimate the distribution of statistics like GLM coefficients from A/B test results, providing estimates of effect size and uncertainty. 3. Bootstrapping models in Spark can be parallelized using multithreading to submit batches of bootstrap iterations concurrently, improving performance by utilizing all CPU cores.
![Analysis of results
• What you often get
• Version B has a statistically significant effect on CR, p = 0.04
• What we ideally want
• Version B increases CR with 92.5% probability
• most likely by 1.8 %, 95% CI: [-0.3; 3.9]
Results of Test 1](https://image.slidesharecdn.com/218ondrejhavlicek-210607232545/85/Bootstrapping-of-PySpark-Models-for-Factorial-A-B-Tests-7-320.jpg)
![Bootstrapping of GLM in Spark
• Multithreading
Worker 1 Worker 2
Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4
Iteration 1 Iteration 2 Iteration 3 Iteration 4
Iteration 5 Iteration 6 Iteration 7 Iteration 8
Iteration 9 Iteration 10 Iteration 11 Iteration 12
... ... ... ...
Batch 1 Batch 2 Batch 3 Batch 4
ret_vals = []
batch_size = math.floor(n_iterations / n_threads)
batches = [{'batchnum': i + 1, 'reps': batch_size} for i in range(n_threads)]
with concurrent.futures.ThreadPoolExecutor(max_workers=n_threads) as executor:
future_run = {
executor.submit(run_batch, df, model, batch['reps']): batch for batch in batches
}
for future in concurrent.futures.as_completed(future_run):
try:
batch_result = future.result()
ret_vals.append(batch_result)
...](https://image.slidesharecdn.com/218ondrejhavlicek-210607232545/85/Bootstrapping-of-PySpark-Models-for-Factorial-A-B-Tests-11-320.jpg)
Bootstrapping of PySpark Models for Factorial A/B Tests
- 1. Bootstrapping of (Py)Spark models for factorial A/B tests Ondrej Havlicek Data Scientist
- 2. Ondřej Havlíček • Senior data scientist • Background • Computer science, psychology, neuroscience • Focus • Inferential statistics, machine learning, ETL • Spark, Python, R • A/B testing, recommendation, search, ... • e-Commerce, social media, ...
- 3. Making data science and machine learning have a real impact on organizations. We are DataSentics PX Personalization for Banking and Insurance DS Innovate AI/ML driven innovation & startups DS TechScale Platforms for AI- intensive applications DS InRetail Improving the customer experience in Retail/FMCG Gold partner & Partner of the Year 2020 Professional partner 4th fastest growing in CE Rising stars award Partners & Awards: Selected Customers: Data science Machine learning specialists Data engineering Cloud specialists 10+ product owners 50+ 30+ Optimize and automate the thousands/millions of small decisions you do everyday Analyse positioning, out-of- stock, pricing and more from a photo. AI choice assistant for e- commerce AI extension for your adform
- 4. Agenda 1. Factorial A/B testing 2. Analysis of results 3. Bootstrapping 4. Performance tuning
- 5. A/B testing • What • A: Control version • B: Experimental version • Why • The only way to improve KPIs consistently • Evidence > HIPPO • Most of tested ideas actually incorrect • How • Usually isolated tests, in parallel or one after another Wikipedia: a user experience research methodology ... consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing ... and determining which of the two variants is more effective.
- 6. Why factorial A/B testing? • Isolated tests are limiting • Few concurrent experiments or very long durations • Solution: Factorial design • Cross multiple tests orthogonally • Each visitor assigned into a variant in all tests • Allows running dozens of simultaneous tests • Each test runs at all traffic • Faster results https://hbr.org/2017/09/the-surprising-power-of-online-experiments
- 7. Analysis of results • What you often get • Version B has a statistically significant effect on CR, p = 0.04 • What we ideally want • Version B increases CR with 92.5% probability • most likely by 1.8 %, 95% CI: [-0.3; 3.9] Results of Test 1
- 8. Analysis of results • How: effect size • Big data: Spark GLM, e.g.: • is_conversion ~ T1 + T2 + T1 * T2 • family = "binomial" • link = "logit" • How: uncertainty • Std. errors generally not provided by Spark GLMs • Bootstrapping • A way to estimate distribution of some statistic • “Poor man’s Bayes”, noninformative prior Results of Test 1
- 9. Bootstrapping • Iterate many times (hundreds..): • Randomly resample data with replacement • Compute statistics of interest: GLM coefficients df_resample = df.sample(withReplacement=True, fraction=1.0) fitted_model = model.fit(df_resample) stats = extract_stats(fitted_model) • How in Spark? • Bootstrapping: Embarrassingly parallel • Spark parallelizes tasks of model fitting = within 1 iteration • How to scale? • Need to run many instances of model fitting in parallel
- 10. Bootstrapping of GLM in Spark in a parallel fashion • Multithreading • Prepare bootstrap iterations into batches: • Each batch contains sequential iterations • Each iteration performs a spark action • Stages have fewer tasks than cores Worker 1 Worker 2 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 6 Iteration 7 Iteration 8 Iteration 9 Iteration 10 Iteration 11 Iteration 12 ... ... ... ... Batch 1 Batch 2 Batch 3 Batch 4 • Submit the batches in parallel using multithreading • Tasks get scheduled in FIFO / FAIR fashion to the executors Iteration 1 Stage 1 Task 1 Task 3 Task 2 Task 4 Core 1 Core 2
- 11. Bootstrapping of GLM in Spark • Multithreading Worker 1 Worker 2 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 6 Iteration 7 Iteration 8 Iteration 9 Iteration 10 Iteration 11 Iteration 12 ... ... ... ... Batch 1 Batch 2 Batch 3 Batch 4 ret_vals = [] batch_size = math.floor(n_iterations / n_threads) batches = [{'batchnum': i + 1, 'reps': batch_size} for i in range(n_threads)] with concurrent.futures.ThreadPoolExecutor(max_workers=n_threads) as executor: future_run = { executor.submit(run_batch, df, model, batch['reps']): batch for batch in batches } for future in concurrent.futures.as_completed(future_run): try: batch_result = future.result() ret_vals.append(batch_result) ...
- 12. Performance: don’t waste resources • How many parallel batches (threads)? • n_threads = n_cores / n_tasks * n_tasks_per_core • n_tasks: repartition to ~100 – 200 MB • n_tasks_per_core: empirical question, ca. 2 – 4 • Check Ganglia UI Worker 1 Worker 2 Core 1 Core 2 Core 3 Core 4 Core 1 Core 2 Core 3 Core 4 Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5 Iteration 6 Iteration 7 Iteration 8 Iteration 9 Iteration 10 Iteration 11 Iteration 12 ... ... ... ... Batch 1 Batch 2 Batch 3 Batch 4
- 13. Performance test
- 14. Lessons learned • Spark better suited for ML than inferential stats • Bootstrapping helps • You can do parallelization^2 in Spark • Business users understand & like the outputs • Core of factorial AB testing is simple • Many interesting challenges in reality J • Overlaps, interactions, funnels, outliers, zero-inflated metrics, variance reduction, ...
- 15. Thank you! Want to know more? Drop me a line ondrej.havlicek@datasentics.com