Applied Machine Learning for Ranking Products in an Ecommerce Setting
5 likes1,817 views
This document outlines a presentation by Arnoud de Munnik and Jerry Vos from Wehkamp on their applied machine learning approach for ranking products in e-commerce. It details their transition to a more flexible and efficient machine learning pipeline using Databricks and discusses the challenges and methodologies involved in product ranking, including feature generation and model evaluation. The goal is to maximize product relevance through advanced ranking algorithms and continuous improvement using A/B testing.
Applied Machine Learning for Ranking Products in an Ecommerce Setting
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Arnoud de Munnik & Jerry Vos, wehkamp Applied Machine Learning for Ranking Products in an Ecommerce Setting #UnifiedDataAnalytics #SparkAISummit
- 3. 3#UnifiedDataAnalytics #SparkAISummit Data Scientist @wehkamp since 2001 Education: Econometrics Jerry Vos Data Scientist @wehkamp since 2011 Education: Marketing Research Arnoud de Munnik
- 4. Agenda • Intro wehkamp • E-commerce ranking problem • Our learning-to-rank pipeline • Ranking model • Q&A 4#UnifiedDataAnalytics #SparkAISummit
- 5. the online department store for families in the Netherlands
- 6. 1952 - first advertisement 1955 - first catalog 1995 - first steps online 2010 - completely online 2018 - mobile first 2019 - a great shop experience our history where we come from
- 7. over 2.000 brands C&A // Vingino // Hunkemöller // Mango // Tommy Hilfiger // Scotch & Soda // ONLY HK Living // House Doctor // Woood // Bloomingville // Zuiver // whkmp’s own our categories Fashion // Home & garden // Electronics // Entertainment // Household // Sports & Leisure // Beauty & Health >400.000 products >500.000 daily visitors 661 million sales 18/19 11 million packages > 950 colleagues 60% of customers shopping mobile 72% of our customers is female
- 8. Our journey 8#UnifiedDataAnalytics #SparkAISummit • We work(ed) with a traditional corporate data warehouse • Need: ML, flexibility, speed, enabling, etc. • 2 years ago: pilot Spark on Databricks – Challenges: Training of people, data in cloud • Today: – Transformation to Databricks / Cloud (S3) – Lots of new (ML) products/prototypes and colleagues on DB platform
- 9. Machine learning @ wehkamp 9#UnifiedDataAnalytics #SparkAISummit Recommend ers Forecasti ng Image classificati on Search Personalisat ion Product ranking Fraud detection And a lot more
- 10. Machine learning @ wehkamp 10#UnifiedDataAnalytics #SparkAISummit Recommend ers Forecasti ng Image classificati on Search Personalisat ion Product ranking Fraud detection And a lot more
- 11. Ranking problem for ecommerce
- 12. Ranking problem for ecommerce 12#UnifiedDataAnalytics #SparkAISummit User search for ‘jeans’ Relevant? We return 4401 products
- 13. Ranking problem for ecommerce 13#UnifiedDataAnalytics #SparkAISummit User navigates to ‘ladies jeans’ overview page Relevant? We return 2176 products
- 14. Ranking problem for ecommerce 14#UnifiedDataAnalytics #SparkAISummit ● Consider a visit to a ‘product overview page’ (example ‘ladies jeans’) as a user query ● Main problem: maximize the order of relevance of returned products given a user query
- 15. ● How good is this list? ● Suppose we know how relevant each item is, can we define an overall score for the relevancy of this list? ● Yes we can, the answer is NDCG (Normalized Discounted Cumulative Gain) https://en.wikipedia.org/wiki/Discounted_cumulative_gain Ranking problem for ecommerce
- 16. ● Suppose we know relevancy scores, let’s rank them ● Let’s add a correction for position via Log2(i+1) ● Divide and sum to get a score: discounted cumulative gain (7,84) ● Do the same, but for this list in perfect order to get an Ideal DCG. That score will be: 9,00 ● Divide our DCG / IDCG = normalized discounted cumulative gain (0.87) Ranking problem for ecommerce 2 3 4 1 i 1 2 1,00 2,00 2 3 1,58 1,89 3 4 2,00 2,00 4 1 2,32 0,43 5 3 2,58 1,16 6 1 2,81 0,36 3 1 Sum: 7,84
- 17. Ranking problem for ecommerce Relevancy scores Explain the scores with features 32 1 Title match 4 Article match Maximize the NDCG, by giving weight to features Reviews Seasonality Price …
- 18. Learning to rank pipeline
- 20. Pipeline 20#UnifiedDataAnalytics #SparkAISummit Data collection 1 Click model 2 Feature generation 3 Ranking model 4 Serve model (ElasticSearch LTR) 5 Evaluation (Tableau) 6 For relevancy scores For explaining relevance For estimating weights to features For productionising
- 21. Efforts 21#UnifiedDataAnalytics #SparkAISummit • Initial effort of building pipeline: 2 data scientists and 1 data engineer (for search and Product Overview Page) for a couple of months • New click/ranking model: 1 data scientist can train, test and push a new ranking model to production within 1 hour
- 22. Data collection 22#UnifiedDataAnalytics #SparkAISummit ● Source: raw Google Analytics feed (daily) ● Per product list (i.e. search, overview page): ○ ProductID ○ Position / Page ○ Impression / Click ● Challenges: ○ tagging is different for web and app ○ devices have different display formats
- 23. Click model Reality: We don’t know the relevancy scores; use a click model. Goal: determine relevance of products in each SOP/POP Approach: predict the relevance of products based on impressions and clicks of products given its position • Clicks over Expected clicks (COEC) • Corrected for small search queries In our case: better results, easier to train & explain • DBN click model (https://github.com/varepsilon/clickmodels) • Paper: Dynamic Bayesian Network ( DBN ) model: Chapelle, O. and Zhang, Y. 2009. A dynamic bayesian network click model for web search ranking. WWW (2009)
- 24. COEC click model 24#UnifiedDataAnalytics #SparkAISummit Example
- 25. COEC click model 25#UnifiedDataAnalytics #SparkAISummit search phrase Product Id clicks Expected clicks Bucket aka relevancy Jeans 0123456 250 50 3 Jeans 6543210 200 20 4 Jeans 3211231 300 300 2 Jeans 4566543 400 800 1 Jeans Random product id 9997979 - - 0Add random data
- 26. Demo clickmodel 26#UnifiedDataAnalytics #SparkAISummit Query: “Flared jeans” Relevancy: 1 Relevancy: 4
- 27. Feature generation Try to explain and predict which attributes (i.e. features) of products (wrt user query) contribute to its relevance score 27#UnifiedDataAnalytics #SparkAISummit - Title match - Description match - Tf-idf - … ● Limit the number of features to < 100 (latency issues) ● For POP features we did not use OHE, but a Bayesian encoder to limit number of features - Popularity - Discount / Promo - Seasonality - Reviews - Days online - Brand - .. Feature examples
- 28. Feature generation 28#UnifiedDataAnalytics #SparkAISummit Feature notebooks Kafka Search processor ES index “jeans” Log results with feature values S3 ● Initial training and query building with snapshot data Query
- 29. Feature generation 29#UnifiedDataAnalytics #SparkAISummit Feature notebooks Delta pre- processed data Delta feature DB Fetch features and send to ES Seasonality estimate per article type OpenWeatherMap API AGG view/sales/promo data Timeseries models with prediction à Scaled via Pandas UDF ES index
- 30. Feature generation 30#UnifiedDataAnalytics #SparkAISummit Feature notebooks Kafka Search processor ES index “jeans” Log results with feature values S3 Query Add clickmodel labels Train model Add model to ES
- 31. Ranking model
- 32. Ranking model • Many machine learning techniques to use • Elastic Search LTR plugin supports XGBoost • XGBoost → eXtreme Gradient Boosting – Variant of the gradient boosting technique (tree-based model) – Non-linearity – Good results (e.g. Kaggle competitions) – Easy to use, tune, and evaluate – Fast (parallel computation on single machine but also cluster support, e.g. Spark) • XGBoost has lots of parameters to tune; we adopt help from Hyperopt https://hyperopt.github.io/hyperopt/ • XGBoost has rank:ndcg as option 32#UnifiedDataAnalytics #SparkAISummit
- 33. Ranking model 33#UnifiedDataAnalytics #SparkAISummit • Each hyperopt run stores the result of its best parameters in MLflow
- 34. Ranking model 34#UnifiedDataAnalytics #SparkAISummit After training, store information in MLflow: • Feature importances (SHAP) • Test and training datasets • Feature map • XGBoost model
- 35. Ranking model 35#UnifiedDataAnalytics #SparkAISummit For examining the feature importance of each model we use SHAP https://github.com/slundberg/shap F1 F2 F3 F4 …
- 36. Serve model 36#UnifiedDataAnalytics #SparkAISummit With a few lines of code we save our model to Elastic index
- 38. Serve model 38#UnifiedDataAnalytics #SparkAISummit Split 1 Split 2 Split 3
- 39. Serve model 39#UnifiedDataAnalytics #SparkAISummit Popularity score > 0.006? NoYes Node 0 Node 1 Node 2 Split 1 Split 2 Split 3 Split 1
- 40. Serve model 40#UnifiedDataAnalytics #SparkAISummit Popularity score > 0.006? No Title match > 9.25? Yes Yes No Node 0 Node 1 Node 2 Node 3 Node 4 Split 1 Split 2 Split 3 Split 1 Split 2
- 41. Evaluation • Use A/B testing to check if ranking models outperform the standard implementation. Configuration of tests done with Planout https://github.com/facebook/planout • An automated Tableau report will show the results of the A/B test • We are reporting quite a few metrics, but most importantly looking at: - Click Trough Rate - Revenue per session - Paul Score https://www.mediawiki.org/wiki/Wikimedia_Discovery/Search/Glossary 41#UnifiedDataAnalytics #SparkAISummit
- 43. Our journey ahead • For search; build multiple models for multiple categories, based on searchphrase classification • Add more product specific attributes • Test with personalisation
- 44. Wrap up 44#UnifiedDataAnalytics #SparkAISummit Automating a learning-to-rank pipeline requires a lot of different parts working together. - Google Analytics - Databricks / Spark - Elasticsearch - S3 - XGBoost - Hyperopt / SHAP - MLflow - Planout - Tableau
- 46. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT