Learning to Rank Datasets for Search with Oscar Castaneda
2 likes1,079 views
Oscar Castañeda presented on learning to rank datasets for search using Elasticsearch's learning to rank capabilities. He discussed extracting relevance judgements during data pipelines to bootstrap a dataset ranking model, and leveraging click-through data on dataset profiles to continuously improve the model. The demo showed ranking movie datasets based on a relevance judgement file, with movie profile pages representing dataset profiles. Next steps included describing datasets in a structured way, building a knowledge graph to extract insights, and using topic models to understand datasets.
![54
Next Steps (1)
• Describe Datasets in a structured schema.org way
using Data Catalog Vocabulary [2].
• Build a knowledge graph and use GraphX to extract insights.
(Useful e.g. for column concept determination (Deng et al. 2013)).
• Build topic models based on structured Datasets using
Glint to perform scalable topic model extraction in Spark
(Jagerman and Eickhoff, 2016) [1].
[1] https://spark-summit.org/eu-2016/events/glint-an-asynchronous-parameter-server-for-spark/
#SAISDS8](https://image.slidesharecdn.com/08oscarcastaneda-181011215158/85/Learning-to-Rank-Datasets-for-Search-with-Oscar-Castaneda-54-320.jpg){kind=icon}
Learning to Rank Datasets for Search with Oscar Castaneda
- 1. Oscar Castañeda, Xoom a PayPal Service Learning to Rank Datasets for Search #SAISDS8
- 2. #SAISDS8 About • Data Scientist at Xoom a PayPal service. • Interests: • Data Management, • Dataset Search, • Learning to Rank. 2
- 3. Spark cluster with Elasticsearch http://bit.ly/2em6RUKhttp://bit.ly/2ebM9HO
- 4. And Indexed RDatasets Spark cluster with Elasticsearch Inside
- 5. 5 Learning to Rank Datasets #SAISDS8
- 6. 6 Learning to Rank Datasets for Search! #SAISDS8
- 7. 7 Agenda • Problem Statement and Motivation • Elasticsearch Learning to Rank • Data Pipeline: metadata extraction, judgement list extraction • Demo: Beginnings of a Dataset Search Engine with Machine- learned relevance ranking. • Q&A #SAISDS8
- 8. 8 Problem Statement (1) • Despite datasets being a key corporate asset they are generally not given the importance they deserve and as a result they are hard to find. #SAISDS8
- 9. 9 Problem Statement (2) • Specifically, teams within organizations have a hard time finding datasets relevant to their function. #SAISDS8
- 11. 11 Topics • Indexing (Spark Summit East 2017). #SAISDS8
- 12. 12 Topics • Indexing (Spark Summit East 2017). • Ranking #SAISDS8
- 13. 13 Topics • Indexing (Spark Summit East 2017). • Ranking => today’s topic! #SAISDS8
- 14. 14 Questions • How are datasets ranked? • Can judgement lists (useful for ranking) be generated at dataset production time? #SAISDS8
- 15. 15 Overview Rdatasets Take ES snapshot Restore ES snapshot http://bit.ly/2e5H1jL #SAISDS8
- 16. 16 Overview Rdatasets Data Pipelines: •Extract Filename, Format, Field description. •Extract Type information. •Index CSV files. Extract Filename Extract Type information. Index CSV files. Extract Format Extract Field description #SAISDS8
- 19. 19 Motivation (1) • Organizing, indexing and ranking Datasets: #SAISDS8
- 20. 20 Motivation (1) • Organizing, indexing and ranking Datasets: • Produced by individual data pipelines • On Data Lake(s) #SAISDS8
- 21. 21 Motivation (2) • Produce a ranking function for datasets that are generated as part of running data pipelines. #SAISDS8
- 22. 22 Motivation (2) • Produce a ranking function for datasets that are generated as part of running data pipelines. • Extract “relevance judgements” and use them to bootstrap a dataset rank model (a posteriori vs. post hoc (Halevy et al., 2016)). #SAISDS8
- 23. 23 Motivation (2) • Produce a ranking function for datasets that are generated as part of running data pipelines. • Extract “relevance judgements” and use them to bootstrap a dataset rank model (a posteriori vs. post hoc (Halevy et al., 2016)). • In a feedback loop leveraging click-through data on dataset profile pages. #SAISDS8
- 24. 24 Organizing, Indexing and Ranking Datasets Index dataset features Input data Data Pipeline Create dataset profile pages. Using Tableau Javascript API ES Cluster Fetch dataset from Datalake and run Data Pipeline on demand. Search Fetch dataset from Datalake. Dataset profile Team Dashboard Feedback logging Relevance judgements #SAISDS8
- 25. 25 Organizing, Indexing and Ranking Datasets Input data Data Pipeline #SAISDS8
- 26. Search 26 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Index dataset features #SAISDS8
- 27. Index dataset features Search 27 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Feedback logging Relevance judgements #SAISDS8
- 28. Index dataset features Search 28 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Feedback logging Relevance judgements #SAISDS8 New
- 29. Index dataset features Search 29 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Feedback logging Relevance judgements #SAISDS8 New New
- 30. Search 30 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Index dataset features Feedback logging Relevance judgements Create dataset profile pages. Dataset profile #SAISDS8
- 31. Search 31 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Index dataset features Feedback logging Relevance judgements Create dataset profile pages. Dataset profile Team Dashboard #SAISDS8
- 32. Search 32 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Index dataset features Feedback logging Relevance judgements Create dataset profile pages. Dataset profile Team Dashboard Fetch dataset from Datalake and run Data Pipeline on demand. Fetch dataset from Datalake. #SAISDS8
- 33. Search 33 Organizing, Indexing and Ranking Datasets Input data Data Pipeline ES Cluster Index dataset features Feedback logging Relevance judgements Create dataset profile pages. Dataset profile Team Dashboard Fetch dataset from Datalake and run Data Pipeline on demand. Fetch dataset from Datalake. #SAISDS8
- 34. 34 How do you rank datasets? #SAISDS8
- 35. Ranking datasets 35 • Extraction of “relevance judgements” can be built into data pipeline for specific datasets immediately after they are generated. #SAISDS8
- 36. Ranking datasets 36 • Extraction of “relevance judgements” can be built into data pipeline for specific datasets immediately after they are generated. • Used to produce a ranking function for datasets. #SAISDS8
- 37. Ranking datasets 37 • Extraction of “relevance judgements” can be built into data pipeline for specific datasets immediately after they are generated. • Used to produce a ranking function for datasets. • Leveraged for training #SAISDS8
- 38. Ranking datasets 38 • Extraction of “relevance judgements” can be built into data pipeline for specific datasets immediately after they are generated. • Used to produce a ranking function for datasets. • Leveraged for training • And to bootstrap a dataset rank model #SAISDS8
- 39. Ranking datasets 39 • Extraction of “relevance judgements” can be built into data pipeline for specific datasets immediately after they are generated. • Used to produce a ranking function for datasets. • Leveraged for training • And to bootstrap a dataset rank model • (a posteriori vs. post hoc (Halevy et al., 2016).) #SAISDS8
- 40. Ranking datasets 40 • Click-through data provides implicit feedback useful to adjust initial relevance judgements. #SAISDS8
- 41. Ranking datasets 41 • Click-through data provides implicit feedback useful to adjust initial relevance judgements. • Leveraging click-through data on dataset profile pages. #SAISDS8
- 42. Ranking datasets 42 • Click-through data provides implicit feedback useful to adjust initial relevance judgements. • Leveraging click-through data on dataset profile pages. #SAISDS8 Create dataset profile pages. Dataset profile
- 43. 43 • Alon et al (2016) advocate finding data in a post- hoc manner by collecting and aggregating metadata after datasets are created or updated. • We propose a so-called “a posteriori” approach where metadata is generated as part of running pipelines using Spark. A posteriori vs. Post-hoc #SAISDS8
- 44. 44 • Alon et al (2016) advocate indexing • We prefer indexing A posteriori vs. Post-hoc #SAISDS8 after the fact immediately after the fact
- 45. 45 Pros • “Relevance judgements” can be extracted and leveraged to bootstrap a ranked dataset index. • In a feedback loop leveraging click-through data on Dataset profile pages. • More granular metrics available to evaluate metadata regeneration. #SAISDS8 immediately after the fact
- 46. 46 Cons • Offline model development is disconnected and only indirectly part of feedback using click-through data. • Looking at trees instead of the forest. • Need to replay indexing pipeline when things change (per data pipeline). #SAISDS8 immediately after the fact
- 47. 47#SAISDS8 Demo!
- 48. 48#SAISDS8 Demo Scenario • Movies represent datasets • TMDB movie pages represent dataset profile pages. • Marketing team (also called WAR team) interested in War movies (datasets).
- 51. 51#SAISDS8 Movie 3 https://www.themoviedb.org/movie/1369-rambo-first-blood-part-ii
- 52. 52#SAISDS8 judgement file Rambo Rambo III Rambo: First Blood Part II
- 53. 53 What have we seen? • How to rank datasets on Elasticsearch using LTR. • Extract relevance judgements immediately after datasets are generated in Spark. • Demo: Dataset Search with Spark and Elasticsearch LTR. #SAISDS8
- 54. 54 Next Steps (1) • Describe Datasets in a structured schema.org way using Data Catalog Vocabulary [2]. • Build a knowledge graph and use GraphX to extract insights. (Useful e.g. for column concept determination (Deng et al. 2013)). • Build topic models based on structured Datasets using Glint to perform scalable topic model extraction in Spark (Jagerman and Eickhoff, 2016) [1]. [1] https://spark-summit.org/eu-2016/events/glint-an-asynchronous-parameter-server-for-spark/ #SAISDS8
- 55. 55 References • Alon Y. Halevy, Flip Korn, Natalya Fridman Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. Goods: Organizing google’s datasets. In Fatmañzcan, Georgia Koutrika, and Sam Madden, editors, Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 795–806. ACM, 2016. ISBN 978-1-4503-3531-7. doi: http://doi.acm.org/10.1145/2882903.2903730. • Katja Hofmann. Fast and Reliable Online Learning to Rank for Information Retrieval. PhD thesis, Informatics Institute, University of Amsterdam, May 2013. • Rolf Jagerman and Carsten Eickhoff. Web-scale topic models in spark: An asynchronous parameter server. CoRR, abs/ 1605.07422, 2016. URL http://arxiv.org/abs/1605. 07422. • Dong Deng, Yu Jiang, Guoliang Li, Jian Li, and Cong Yu. Scalable column concept de- termination for web tables using large knowledge bases. PVLDB, 6(13):1606–1617, 2013. doi: http://www.vldb.org/pvldb/vol6/p1606-li.pdf. • Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. Multileave gradient descent for fast online learning to rank. In WSDM 2016: The 9th International Conference on Web Search and Data Mining, pages 457-466. ACM, February 2016. • Sreeram Balakrishnan, Alon Y. Halevy, Boulos Harb, Hongrae Lee, Jayant Madhavan, Afshin Rostamizadeh, Warren Shen, Kenneth Wilder, Fei Wu 0003, and Cong Yu. Ap- plying webtables in practice. In CIDR 2015, Seventh Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 4-7, 2015, Online Proceedings. www.cidrdb.org, 2015. #SAISDS8
- 56. 56 Q&A #SAISDS8
- 57. Thank You. Email: ocastaneda@paypal.com Twitter: @oscar_castaneda #SAISDS8