Measuring Search Engine Quality using Spark and Python
Download as PPTX, PDF6 likes1,946 views
The document discusses the migration of Elsevier's ScienceDirect search engine from Microsoft FAST to Apache Solr while maintaining search quality. It outlines the development of a tool to measure search engine quality using query and click logs, which allows for on-demand assessment without requiring production deployment. The results showed that Solr PDF downloads were nearly equivalent to FAST downloads, leading to the decision to implement Solr in production.
Measuring Search Engine Quality using Spark and Python
- 1. Sujit Pal March 13, 2016 Measuring Search Engine Quality using Spark and Python
- 2. | 2 ⢠About Me ď§ Work at Elsevier Labs ď§ Interests: Search, NLP and Distributed Processing. ď§ URL: labs.elsevier.com ď§ Email: sujit.pal@elsevier.com ď§ Blog: Salmon Run ď§ Twitter: @palsujit ⢠About Elsevier ď§ Worldâs largest publisher of STM Books and Journals ď§ Uses data to inform and enable consumers of STM info Introduction
- 3. | 3 ⢠Problem Description ⢠Our Solution ⢠Other Uses ⢠Future Work ⢠Q&A Agenda
- 5. ⢠Migrate the Science Direct Search Engine from Microsoft FAST to Apache Solr. Background
- 6. ⢠Keep search quality consistent across platforms. ⢠Full text downloads from Solr must equal or exceed that from FAST. Success Criteria
- 8. ⢠A/B Tests happen in Production ď§ Expensive: Requires Production Deployment of Search Engine(s). ď§ Risky: Bad customer experience in AB can result in customer leaving. ď§ Limited scope for iterative improvement: because of expense and risk. ď§ A/B tests take time to be statistically meaningful. ⢠We needed something that ď§ Could be run by DEV/QA on demand. ď§ Produces repeatable indicator of search engine quality. ď§ Does not require production deployment. ⢠This tool is the subject of our talk today. ButâŚ
- 9. Our Solution
- 10. ⢠We have ď§ Query Logs â from query string entered by user. ď§ Click Logs â from the download links clicked by user. ⢠We can generate ď§ Search results for each query against the search engine. ⢠Combining which we can provide ď§ Click Rank Distribution for a search engine (configuration) Solution Overview
- 11. Click Rank Definition ⢠Click Rank is the sum of the ranks of all PIIs in the result set that match the PIIs clicked for that query, divided by the number of matches. ⢠Deconstructing the above: ď§ Let the click logs for a query be the document set Q. ď§ Let the top N search results for the query be represented by a List R. ď§ Let P be the intersection of Q and R, and P be the (one-based) indexes of the documents in R that are in P. ď§ Click Rank = ÎŁ P / || P ||
- 12. Inputs
- 13. Preprocess Logs ⢠Query and Click logs already merged by A/B test framework. ⢠Provided as line-oriented JSON format.
- 14. Generate Search Results for Engine ⢠Replay query logs against Search Engine (configuration). ⢠Use Python Multiprocessing module (for parallel access).
- 15. Search Result Generation Code
- 16. Search Results Example Data ⢠Search Results are saved one file per query. ⢠Top 50 results extracted (so each file is 50 lines long). ⢠Maintains parity with FAST reference query results (provided one- time via legacy process).
- 17. Compute Click Rank Distribution ⢠Use Apache Spark to compute the Click Rank Distribution. ⢠Use Python + Matplotlib to build reports and visualizations.
- 18. Spark Code for Generating Click Rank Distribution ⢠Skeleton of a PySpark Program
- 19. Spark Code for Generating Click Rank Distribution ⢠Step #1: Convert the Clicks data to (Query_ID, List of clicked PIIs)
- 20. Spark Code for Generating Click Rank Distribution ⢠Step #2: Convert the Search Results data to (Query_ID, List of PIIs in search result)
- 21. Spark Code for Generating Click Rank Distribution ⢠Step #3: Join the two RDDs and compute Click Rank from the intersection of the clicked PIIs and the result PIIs for each Query_ID.
- 22. Generate Reports ⢠Download Click Rank Distribution for Search Engine (configuration). ⢠Use Python + Matplotlib to build reports and visualizations.
- 23. Outputs from Tool ⢠Step #4: Download distribution from S3 and aggregate to chart and spreadsheet.
- 24. How did we do (in our A/B test)? ⢠Solr PDF downloads were 99.6% of FAST downloads. ⢠Difference in download rates not statistically significant. ⢠Decision made to put Solr into production. 90% 95% 100% 105% Jan (AB #1) Feb (AB #2) Mar (AB #3) Apr (AB #4) SOLR Downloads as % of FAST Downloads PDF HTML Level of FAST Downloads
- 25. Other Uses
- 26. Find Search Result Overlap between Configurations ⢠Measure drift between two search configurations. ⢠Ordered and Unordered comparison at different top N positions. ⢠Result set overlap increases with N. ⢠Lot of positional overlap in the top N positions across engines.
- 27. Search Quality as Overlap between Title and Query ⢠Measures overlap of title words with query words for various top N positions. ⢠Overlap @ N defined as sum of number of words overlap for the first N titles with the query normalized by N times the number of words in the query. ⢠Overlap @ N decreases monotonically with N. ⢠Solr engines seem to do better at this measure.
- 28. Click Distribution ⢠Measures the distribution of clicked positions across the top 50 positions for each engine and compares them. ⢠In this chart, FAST has higher number of clicks at the top positions than the Solr configurations shown.
- 29. Distribution of Publication Dates in Results ⢠The engine has a temporal component in its ranking algorithm. ⢠Compares the distribution of publication dates across search engine configurations to visualize its behavior.
- 30. More Uses ⌠⢠Measuring impact of query response time on click rank. ⢠Comparing click rank distributions by document type. ⢠âŚ
- 31. ⢠Compute (Average/Median) CR per user. ⢠Compute CR per query and user. ⢠Use this as input to Learning to Rank Algorithms. ⢠Other ideas⌠Future Work
- 32. Thank you for listening! Questions? My Email: sujit.pal@elsevier.com