GOTO Night: Decision Making Based on Machine Learning
0 likes141 views
The document describes the evolution of a decision-making system called 'smart gateway' that integrates input from product managers, data scientists, and software engineers to enhance website personalization and business processes. It highlights the complexities involved in creating a production-ready platform that supports real-time data access, configurable experiments, and decision points. Ultimately, the collaborative efforts resulted in a powerful environment capable of running various algorithms, paving the way for future innovations.
GOTO Night: Decision Making Based on Machine Learning
- 1. Decision Making Based on Machine Learning at . Jesper Richter-Reichhelm (@jrirei)
- 2. In the beginning there was void and every customer was seeing the same version of the website.
- 3. But then came along a Product manager and spoke:
- 4. “What is this? I need to be able to test variants of our products. I will introduce something!”
- 5. And she took an off-the-shelf solution and now every customer could see a different version of the website. And after it was done she was happy and life was good.
- 6. But then came along a Data Scientist and spoke:
- 7. “What is this? I need to be able to add complex algorithms to steer business processes. I will take this and make it better!”
- 8. And he hacked together wrote a complex decision system that he would call Smart Gateway because it could steer processes throughout all the company. And after it was done both were happy and life was good.
- 9. But then came along a Software Engineer and spoke:
- 10. “What is this? If there would come just one strong wind, everything would fall down and crash. I will take this and make it better!”
- 11. And he would take the Smart Gateway and rewrite it from scratch refactor it a bit to make it production ready. And after it was done all three were happy and life was good.
- 12. But then all three looked at the thing that they have created together and spoke:
- 13. “What is this? How did this become something so much more powerful than what we set out to do in the first place?”
- 14. “Actually we have truly build a …”
- 15. Relevant fashion for men
- 17. Day 1
- 18. Day 1 Day 3
- 19. Day 1 Day 3 Day 4
- 20. Day 1 Day 3 Day 4 Day 5
- 21. Customer journey is quite “long” Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold
- 22. Order economics demand complex tradeoffs Cart Cost Goods kept Returned
- 23. Order economics demand complex tradeoffs Cart Cost operational profit Goods kept Buying Logistics Service Other
- 24. Science = Data & Tech Art = Stylist team King = Customer
- 25. Science = Data & Tech Art = Stylist team Cart Cost Data Driven Vision
- 26. AB Tests Chapter 1 driven by Product
- 27. AB Tests with an external service
- 28. AB Tests with an external service
- 29. Canary testing with nginx
- 30. Canary testing with nginx
- 32. Problem: Parallel experiments with canary tests A B
- 33. Problem: Parallel experiments with canary tests AB BAAA BB
- 34. Problem: External tool can only track conversion
- 35. Problem: External tool can only track conversion Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold
- 36. Problem: External tool does not support decision points Cart Cost operational profit Goods kept Buying Logistics Service Other
- 37. Problem: External tool does not support decision points
- 38. Smart Gateway Chapter 2 driven by Data Science
- 39. Goals • Integration with IT platform • Decision Points yield persistent results • Complex decision making logic to allow process steering • Configurable for flexibility
- 40. Smart Gateway is integrated in platform Service Service Smart Gateway Service REST Black Box
- 41. Smart Gateways have persistence Service Service Smart Gateway Service Black Box SQL
- 42. Smart Gateways are “smart” • Data Scientists create an algorithm, called Decider • Algorithm can have DWH access • Algorithm is packed in a Docker container and listens to http calls
- 43. Smart Gateways are “smart” Service Service Smart Gateway Service Black Box SQL
- 44. Smart Gateways are “smart” Service Service Smart Gateway Service Decider Decider Decider DWH http SQL
- 45. Smart Gateways are configurable • Sampling: routing of traffic to experiments • Experiments can opt in independently • Experiments define groups (random distribution) • Groups map to a Decider
- 46. Smart Gateways are configurable Decider Decider Decider Decider Decider Decider Sampling Experiment Experiment 25% 50% of “DE” default “A” “B” Age < 30 ? “A” : “B” Age < 35 ? “A” : “B” <something smart> <some default> 50% 50% 50% 30% 20%
- 48. Problem: Data is not real time and can’t be consistent Service Service Smart Gateway Service Decider Decider Decider DWH data import SQL
- 49. Problem: One single point of failure Service Service Smart Gateway Service Decider Decider Decider DWH SQL all on one host default decision
- 50. Smart Gateway v2 Chapter 3 driven by IT
- 51. Goals • Separation of concerns • Data access is real time • Production ready setup
- 52. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data / ML BI
- 53. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data Delivery Data / ML BI
- 54. Created a new team to shift responsibilities Service Service Smart Gateway Service Decider Decider Decider DWH SQL
- 55. Configuration is now DB based • It simply got too big • Decision Point • n Experiments per Decision Point • n Deciders per Experiment • REST API to change config on the fly
- 56. Smart Gateways do “data enrichment” • Deciders become pure functions • Before sampling of traffic, call is enriched with data • Enrichment is configurable • Enriched data can be used to decide if experiments “opt in”
- 57. Smart Gateways now do “data enrichment” Enrichment Sampling Experiment Decider Decider DeciderRetrieve data: • REST call • constant value • LocalStore (K/V) • JS Script
- 58. Smart Gateways become production ready • Professional Docker setup • Orchestrating Docker containers via Kubernetes
- 59. Decider Decider Decider Smart Gateway Smart Gateways become production ready Service Service Smart Gateway Service Decider Decider Decider SQL Service Service SQL Redis Data Job
- 60. Example use cases • AB testing • Decision Points • User discrimination to prepayment • … and many more • Also very nice: shadow mode
- 62. Smart Gateways Service Service Smart GatewaySmart Gateway Service DeciderDecider DeciderDecider DeciderDecider Service Service RedisSQL Data Job
- 63. Smart Gateways are a runtime environment for pure functions Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job
- 64. Delivery platform for pure functions Chapter 4 driven by Product, Data Science, and IT
- 65. Our own delivery platform for algorithms
- 66. Example 1: Supporting our stylists in decision making Filters… Customer Info Order / Basket Available Stock • For every order • For every article in stock • Calculate a score how likely customer is to keep this article • Score is used to encourage or discourage stylist to pick an article
- 67. Example 2: Scanning return slips • External partner would take too long • Implemented algorithm on our own • Every scan is handled via Smart Gateway • Information on follow on orders is now available
- 68. Filters… Customer Info Order / Basket Available Stock Smart Gateway 120 sec execution time3 MB data (input / output)
- 69. Smart Gateway became asynchronous (if necessary) Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job
- 70. AWESOME
- 71. Conclusion By me
- 72. But then all three looked at the thing that they have created together and spoke:
- 73. “What is this? How did this become something so much more powerful than what we set out to do in the first place?”
- 74. “Actually we have truly build a system that can run just any pure function. AB tests and process steering are just the beginning. Maybe speech recognition is next. Or maybe…”
- 75. With that they started to brainstorm all the cool things they could do in the future.
- 76. Time and the next Hackathon will tell what they made out of it. And maybe I can come back next year to tell you all about it.
- 77. The End