How we integrate Machine Learning Algorithms into our IT Platform at Outfittery
1 like1,636 views
1) Over time, a product manager, data scientist, and software engineer each contributed to building a machine learning platform at a company to improve customer experiences and business processes. 2) They created a "Smart Gateway" system to run algorithms and experiments, but it had limitations around data consistency and scalability. 3) A new IT team took over the system and enhanced it with real-time data access, production readiness using Docker and Kubernetes, and a configuration database. 4) The system evolved to become a general platform for running "pure functions" to power applications beyond just experiments, like stylist recommendations and return form processing.
How we integrate Machine Learning Algorithms into our IT Platform at Outfittery
- 1. Jesper Richter-Reichhelm (@jrirei) How we integrate Machine Learning Algorithms into our IT Platform BI & Marketing - Architecture … with annotations! … with annotations! … with annotations! … with annotations!
- 2. In the beginning there was void and every customer was seeing the same version of the website. A quick trailer for the talk The whole content in just 2 minutes!
- 3. But then came along a Product manager and spoke:
- 4. “What is this? I need to be able to test variants of our products. I will introduce something!”
- 5. And she took an off-the-shelf solution and now every customer could see a different version of the website. And after it was done she was happy and life was good.
- 6. But then came along a Data Scientist and spoke:
- 7. “What is this? I need to be able to add complex algorithms to steer business processes. I will take this and make it better!”
- 8. And he hacked wrote a complex decision system that he would call Smart Gateway because it could steer processes throughout all the company. And after it was done both were happy and life was good.
- 9. But then came along a Software Engineer and spoke:
- 10. “What is this? If there would come just one strong wind, everything would fall down and crash. I will take this and make it better!”
- 11. And he would take the Smart Gateway and rewrite it from scratch refactor it a bit to make it production ready. And after it was done all three were happy and life was good.
- 12. But then all three looked at the thing that they have created together and spoke:
- 13. “What is this? How did this become something so much more powerful than what we set out to do in the first place?”
- 14. “Actually we have truly build a …” … read the whole talk to find out! :-) Hey, that was just a trailer…
- 15. Relevant fashion for men Short commercial after the trailer…
- 16. Outfittery’s business model: Customer - a man - orders a box. He may or may not schedule a talk to a stylist.
- 17. Day 1 Stylists pack an individual box
- 18. Day 1 Day 3 Box is sent to the customer
- 19. Day 1 Day 3 Day 4 Customer is happy to unpack the box
- 20. Day 1 Day 3 Day 4 Day 5 … but is too busy to send his return back quickly
- 21. Customer journey is quite “long” Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold
- 22. Order economics demand complex tradeoffs Cart Cost Goods kept Returned
- 23. Order economics demand complex tradeoffs Cart Cost operational profit Goods kept Buying Logistics Service Other
- 25. Science = Data & Tech Art = Stylist team King = Customer
- 26. Science = Data & Tech Art = Stylist team Cart Cost Data Driven Vision
- 27. AB Tests Chapter 1 driven by Product
- 28. AB Tests with an external service It’s a good service, easy to setup and use.
- 29. AB Tests with an external service But it’s focused on conversion.
- 30. Canary testing with nginx For more complex ab tests …
- 31. Canary testing with nginx … we deploy multiple version of the website.
- 32. It works, but…
- 33. Problem: Parallel experiments with canary tests A B This is nice for a single test…
- 34. Problem: Parallel experiments with canary tests AB BAAA BB … but it’s hell to run multiple tests in parallel!
- 35. Problem: External tool can only track conversion This works well for conversion tests…
- 36. Problem: External tool can only track conversion Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold … but it fails when faced with our long lead times.
- 37. Problem: External tool does not support decision points Cart Cost operational profit Goods kept Buying Logistics Service Other … but it fails when handling different KPIs.
- 38. Problem: External tool does not support decision points Also: We not only do AB tests, … we want decision points to steer processes.
- 39. Smart Gateway Chapter 2 driven by Data Science
- 40. Goals • Integration with IT platform • Decision Points yield persistent results • Complex decision making logic to allow process steering • Configurable for flexibility
- 41. Smart Gateway is integrated in platform Service Service Smart Gateway Service REST Black Box Implemen- tation handled as black box On the left: responsibility of IT (CTO) On the right: responsibility of Data (CDO)
- 42. Smart Gateways have persistence Service Service Smart Gateway Service Black Box SQL
- 43. Smart Gateways are “smart” • Data Scientists create an algorithm, called Decider • Algorithm can have DWH access • Algorithm is packed in a Docker container and listens to http calls
- 44. Smart Gateways are “smart” Service Service Smart Gateway Service Black Box SQL
- 45. Smart Gateways are “smart” Service Service Smart Gateway Service Decider Decider Decider DWH http SQL Inside the “black box”
- 46. Smart Gateways are configurable • Sampling: routing of traffic to experiments • Experiments can opt in independently • Experiments define groups (random distribution) • Groups map to a Decider … all in one gigantic YAML file. :-)
- 47. Smart Gateways are configurable Phase 1: Sample traffic to experiments Phase 2: Treatment by a Decider
- 48. Worked nice for about 1 year, but …
- 49. Problem: Data is not real time and can’t be consistent Service Service Smart Gateway Service Decider Decider Decider DWH data import SQL Break consistency and not applicable for real time Data imports are periodic batch jobs.
- 50. Problem: One single point of failure Service Service Smart Gateway Service Decider Decider Decider DWH SQL all on one host default decision … which essentially breaks the “black box”. This forces clients to provide default values
- 51. Smart Gateway v2 Chapter 3 driven by IT
- 52. Goals • Separation of concerns • Data access is real time • Production ready setup
- 53. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data / ML BI
- 54. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data Delivery Data / ML BI A new IT team… … sitting “within” the Data department.
- 55. Created a new team to shift responsibilities Service Service Smart Gateway Service Decider Decider Decider DWH SQL New team took over Smart Gateway (dev + ops)
- 56. Configuration is now DB based • It simply got too big • Decision Point • n Experiments per Decision Point • n Deciders per Experiment • REST API to change config on the fly Data dep. can update config / deciders “on the fly”. Bye bye gigantic YAML file!
- 57. Smart Gateways do “data enrichment” • Deciders become pure functions • Before sampling of traffic, call is enriched with data • Enrichment is configurable • Enriched data can be used to decide if experiments “opt in” Data consistency and real time can be guaranteed. IT is now responsible for providing data:
- 58. Smart Gateways now do “data enrichment” Enrichment Sampling Treatment Decider Decider DeciderRetrieve data: • REST call • constant value • LocalStore (K/V) • JS Script Routing to experiment: • filtering possible • “once only” possible Group assignment: • defined group size • random selection Added a new “data enrichment” phase
- 59. Smart Gateways become production ready • Professional Docker setup • Orchestrating Docker containers via Kubernetes
- 60. Decider Decider Decider Smart Gateway Smart Gateways become production ready Service Service Smart Gateway Service Decider Decider Decider SQLRedis Service Service SQL
- 61. Example use cases • AB testing • Decision Points • User discrimination to prepayment • … and many more
- 62. Current status: We are happy!
- 63. Smart Gateways Service Service Smart GatewaySmart Gateway Service DeciderDecider DeciderDecider DeciderDecider Service Service RedisSQL Data Job But we can do more:
- 64. Smart Gateways are a runtime environment for pure functions Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job Pure functions can do more than just “decide”!
- 65. Delivery platform for pure functions Chapter 4 driven by Product, Data Science, and IT
- 66. Our own delivery platform for algorithms Ok: AWS Lambda is more powerful but our solution is deeply integrated.
- 67. Example 1: Supporting our stylists in decision making AI helping stylists to make better decisions Sorry, no real screenshots in the public. :-)
- 68. Example 1: Supporting our stylists in decision making • For every order • For every article in stock • Calculate a score how likely customer is to keep this article • Score is used to encourage or discourage stylist to pick that article Output is a scoring of articles for that specific box
- 69. Example 1: Supporting our stylists in decision making
- 70. Example 2: Scanning return slips Closing a process gap - fast!
- 71. Example 2: Return slip scans • External partner would take too long • Implemented algorithm on our own • Every scan is handled via Smart Gateway • Information on follow on orders is now available Output: Which checkboxes were checked
- 72. Smart Gateway became asynchronous (if necessary) Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job These “jobs” needed async handling: We use RabbitMQ and callbacks.
- 73. Smart Gateway became asynchronous (if necessary) 60 sec execution time only 5 sec execution time but 10 MB data (input / output) Jobs with long execution time or quite big data reqs can be handled now.
- 74. Conclusion By me
- 75. But then all three looked at the thing that they have created together and spoke: The talk was held at a cinema so let’s stick with the theme.
- 76. “What is this? How did this become something so much more powerful than what we set out to do in the first place?”
- 77. “Actually we have truly build a system that can run just any pure function. AB tests and process steering are just the beginning. Maybe speech recognition is next. Or maybe…”
- 78. With that they started to brainstorm all the cool things they could do in the future.
- 79. Time and the next Hackathon will tell what they made out of it. And maybe I can come back next year to tell you all about it.
- 80. The End
- 81. Questions? Jesper Richter-Reichhelm (@jrirei) Feel free to contact me if you have questions!