Running Stan in Production: Bayesian Revenue Estimation
0 likes636 views
The document discusses the implementation of Bayesian revenue estimation using Stan in a production environment by Smartly.io, focusing on campaign budget allocation and the challenges faced in modeling results. It outlines the transition from an existing model with significant data variations to a new hierarchical model utilizing log-normal distributions for revenue estimation. Key takeaways include the importance of simplifying models for production stability and employing advanced techniques like ADVI to improve inference efficiency.
Running Stan in Production: Bayesian Revenue Estimation
- 1. Running Stan in Production: Bayesian Revenue Estimation 2018-08-29 StanCon Markus Ojala PhD, Chief Data Scientist Smartly.io @MarkusOjala
- 2. Implemented in early 2017: https://www.smartly.io/blog/tutorial-how-we-productized-bay esian-revenue-estimation-with-stan Referred by Andrew Gelman in early 2018: http://andrewgelman.com/2018/01/21/smartly-io-productize d-bayesian-revenue-estimation-stan/ “It’s BDA come to life!“ “This sort of thing is exactly what we were hoping to see.” Blog posts
- 3. 1 billion spend Managed yearly We Make Online Advertising Easy, Effective, And Enjoyable Facebook & Instagram Partner
- 5. Use Case: Campaign Budget Allocation
- 7. The number of pulls for a given lever should match its actual probability of being the optimal lever Sample from the posterior for the mean of each lever Bayesian Bandits / Thompson sampling
- 8. Modeling results per spend
- 9. Separate revenue model into two parts ROAS = revenue / cost = revenue / conversions * conversions / cost = revenue / conversions * 1 / CPA ROAS = return on ad spend CPA = cost per action Existing model ● Lot of data ● Varies fast ● Big differences New model ● Little data ● Varies slowly ● Small real differences ● Lot of random variation
- 10. ● Use hierarchical model: Account -> Campaign -> Ad set ● Revenue follows usually a long-tailed distribution, use log-normal ● We observe only hourly aggregates: approximate by using Fenton-Wilkinson log-normal approximation Modeling the revenue per purchase
- 11. Stan!
- 13. Posterior predictive checks with +100 real data
- 14. Fix step-by-step, allow no exceptions
- 16. ● Easy way to write Bayesian models and do inference ● But hard to get stable in production: limit the scope of the model ● Reparametrize, use reasonable informative priors and custom initialization ○ Solves most of the convergence issues ● Sampling does not scale: we use ADVI / variational inference approximation ○ Validate that you get same results with sampling and ADVI ○ Random initialization issues were solved by fixing ADVI parameters ● About 1000 daily runs and $1M scheduled with Celery, PyStan, monitoring Learnings in productizing Stan
- 17. Results 1 Common case: ad sets don’t really differ
- 18. Results 2 Campaigns differ, ad sets not
- 19. Results 3 Lot of data Ad sets differ
- 20. Thank you. Markus.Ojala@smartly.io Are you an expert in Bayesian modelling? Join us! https://www.smartly.io/careers
- 21. We observe only aggregates Goal: estimate log-normal parameters for ad sets Challenge: observation i is aggregate of multiple events ni Solution: Estimate by another log-normal