"How did I learn how not to run a Data Science project" by Florian Bonnet
Download as PPTX, PDF0 likes228 views
The document discusses HelloFresh's operational model, emphasizing their efficient, waste-reducing supply chain and successful referral marketing strategies. It also highlights the analysis of a failed A/B testing process, identifying issues such as overconfidence in results, inadequate data tracking, and lack of clear ownership. Key learnings include the importance of involving all stakeholders early, ensuring accountability, and maintaining continuous quality monitoring.
"How did I learn how not to run a Data Science project" by Florian Bonnet
- 1. Florian Bonnet - HelloFresh August 20th 2019
- 2. 1. Starters - HelloFresh – who we are/what we do? 2. Entrée – Referral Marketing - A channel prone to optimization 3. Main Dish – Back to the Future – Dissecting a Failure 4. Dessert – Summary – What did I learn
- 3. HelloFresh Who we are? What we do?
- 4. Spicies ... ... ... ... ... Oct. 11 Oct. 18 Oct. 25 Nov. 1 Nov. 8 Nov. 15 Freshly sourced ingredients in the right amount Delivered to your home on a weekly basis Subscription model but you can pause anytime
- 5. TYPICAL FOOD SUPPLY CHAIN 10 days and 5 parties OUR SUPPLY CHAIN REVOLUTION 3 days and 3 parties Consumer Home Day 10 23% waste Producer Day 1 Wholesaler Day 2 2% waste Warehouse Day 4 3% waste Supermarket Day 6 11% waste STARTING WITH CONSUMER FASTER AND FRESHER NO FOOD WASTE Better for Customers More Margin for HelloFresh Consumer Home Day 3 Producer Day 1 Consume r Day 2 Source: Company information; United States Department of Agriculture; Canaccord Genuity estimates Note: 5% “Farm to Retail” waste data split to wholesale and warehouse.
- 6. 1. Source: Recent company filings 2.5m active users Best Meal Delivery Service #1 Industrial Excellenc e Award DE&EU
- 7. Referral Marketing An Important marketing channel prone to optimization
- 8. Growth of Customer Base Paid Marketing Customer Base Boxes Shipped Marketing Manager CCV/CAC = Low Unhappy Manager
- 9. Growth of Customer Base Paid Marketing Customer Base Boxes Shipped Marketing Manager What‘s up! Check out HelloFresh it is super cool! Good Thx :) Awesome!!! Customer Friend More Customers High + Happy Manager
- 11. Customer Friend Free ... ... Profitability threshold 1 #boxes CCV/CAC
- 12. Customer Friend Free ... ... Profitability threshold 1 #boxes CCV/CAC
- 13. Customer Friend Free ... ... Profitability threshold 1 #boxes CCV/CAC
- 14. Alles gut Customer Friend Free ... ... Profitability threshold 1 #boxes CCV/CAC
- 15. Customer Friend Free Free Free Free Free Free Free Find a way for customer to order more Identify bad referrer upfront and prevent them from referring 1 2
- 16. Friend checkout
- 17. Friend checkout Surprised 2nd box discount at checkout
- 18. Friend checkout Surprised 2nd box discount at checkout Discount $10 $20 $30 $40 Uplift 52W LTV -2.3% +2.1% +1.4% +8.2% Phase1 Randomized AB Test POC Data Gathering
- 19. Friend checkout Surprised 2nd box discount at checkout Discount $10 $20 $30 $40 Uplift 52W LTV -2.3% +2.1% +1.4% +8.2% Phase1Phase2 Randomized AB Test POC Data Gathering Data Science Modeling Finding best discount for each friend Apply in production
- 20. Friend checkout Surprised 2nd box discount at checkout Discount $10 $20 $30 $40 Uplift 52W LTV -2.3% +2.1% +1.4% +8.2% Phase1Phase2 Randomized AB Test POC Data Gathering Data Science Modeling Finding best discount for each friend Apply in production Customer Friends Behavioral Data Revenue/costs Data
- 21. Friend checkout Surprised 2nd box discount at checkout Discount $10 $20 $30 $40 Uplift 52W LTV -2.3% +2.1% +1.4% +8.2% Phase1Phase2 Randomized AB Test POC Data Gathering Data Science Modeling Finding best discount for each friend Apply in production Customer Friends Behavioral Data Revenue/costs Data Customer Free time #1 Free #2 Free #3 Free #4 Try it!Really good! Changed my life Best Recipes!
- 22. Friend checkout Surprised 2nd box discount at checkout Discount $10 $20 $30 $40 Uplift 52W LTV -2.3% +2.1% +1.4% +8.2% Phase1Phase2 Randomized AB Test POC Data Gathering Data Science Modeling Finding best discount for each friend Apply in production Customer Friends Behavioral Data Revenue/costs Data Customer Free time #1 Free #2 Free #3 Free #4 Try it!Really good! Changed my life Best Recipes!
- 23. Friend checkout Surprised 2nd box discount at checkout Phase1Phase2 Randomized AB Test (Q1) POC Data Gathering Data Science Modeling Finding best discount for each friend Apply in production Customer Friends Behavioral Data Revenue/costs Data Customer Free time Friend#1 Free Friend#2 Free Friend#3 Free Firend#4 Try it!Really good! Changed my life Best Recipes! Which discount for which friend ot maximize cohort ROI? CustomerCustomerCustomerCustomer
- 24. 1 2 3 Display the right discount at checkout Store the discount on Friend‘s account Track Redemptions
- 25. 1 year of work Multiple fails in execution and process Lots of rework No tangible results Frustrated team Frustrated stakeholders $0 in the bank No knowledge aquired
- 26. Back to the Future Dissecting a failure
- 27. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
- 28. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2 Too confident in results No deep dive in data and User Experience to see if results reflected the experience proposed: ‒ Not all test users exposed to the popup ‒ Unbalanced test and control groups
- 29. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2 Too confident No deep dive Requirements not clear enough ‒ Experience not well thought through leading to several changes in scope No clear ownership / project manager ‒ Each function thinking the other ones do the necessary holes in the net resulting in last minute rework or missing pieces No proper tracking ‒ Experience and key elements not tracked properly dirty data
- 30. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2 Too confident No deep dive Requirements not clear enough No clear ownership / project manager No proper tracking Confusions Data Science / Data Analysts slow down process Analysis not prepared enough ahead of time loss of time No clear expectations as of the results between DS, PO, stakeholders doubt spreading around
- 31. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2 Too confident No deep dive Requirements not clear enough No clear ownership / project manager No proper tracking Confusions Data Science / Data Analyst Analysis not prepared enough ahead of time No clear expectations as of the results Creation of heavy machinery to solve all the issues huge amount of resources mobilized & NSMMVP Changes in requirements from Data Science to make it a success frustration, complexification, lack of trust, lost time No clear expectations on results and next steps
- 32. A/B test And decision Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2 Too confident No deep dive Requirements not clear enough No clear ownership / project manager No proper tracking Confusions Data Science / Data Analyst Analysis not prepared enough ahead of time No clear expectations as of the results Creation of heavy machinery to solve all the issues Changes in requirements from Data Science to make it a success No clear expectations on results and next steps
- 33. Summary What did I learn?
- 34. BE FE QA PO DS PAML Involve everyone from day one BE PO PA DS QA ML 1
- 35. Clear PMO (does not have to be the PO) Accountable for success 2
- 36. Customer Friends Behavioral Data Revenue/costs Data DSPA Actionable A/B Test Quick iteration to validate model and process Simple Model first vs. Nobel prize machinery 3
- 37. A/B test QA, QA, QA, QA a.k.a monitor the experience continuously 4
- 38. Don’t be afraid to cut your losses 5
- 39. DSPA Actionable A/B Test Quick iteration to validate model and process Simple Model first vs. Nobel prizemachinery Clear PMO accountable for success Involve everyone from day one BE FE QA PO DS PAML 1 2 3 A/B test 4 5 Agile all the way Don’t be afraid to cut your losses QA,QA,QA,QA