Search Accuracy Metrics and Predictive Analytics - A Big Data Use Case: Presented Paul Nelson, Search Technologies
2 likes1,099 views
This document describes a new approach to evaluating search engine accuracy using predictive analytics and big data. The key points are: - It presents a method to reliably measure and compare search engine accuracy offline using query logs and click logs, without requiring deployment to production. - It analyzes activity at the user and session level to understand individual search behaviors and calculate engine scores based on relevance to each user. - Leveraging big data, it uses a statistical model trained on past query and click data to predict the probability of relevance for new results, providing a more objective scoring method. - This predictive relevance scoring approach identifies important parameters and allows experimenting to continuously improve search engine performance over time based on data and science
![10
Engine Score
• Group activity by session and/or user (Queries & Clicks)
• Determine “relevant” documents
o What did the user view? Add to cart? Purchase?
o Did the search engine return what the user ultimately wanted?
• Determine engine score per query based on user’s POV
o Σ power(FACTOR, position)*isRelevant[user, searchResult[position].DocID]
o (Note: many other formulae possible, MRR, MAP, DCG, etc.)
• Average score for all user queries = user score
• Average scores across all users = final engine score](https://image.slidesharecdn.com/searchaccuracymetrics-paulnelson-151021174545-lva1-app6891/85/Search-Accuracy-Metrics-and-Predictive-Analytics-A-Big-Data-Use-Case-Presented-Paul-Nelson-Search-Technologies-10-320.jpg)
![12
Off-Line Engine Analysis
o Can we re-compute this array for all queries?
o ANSWER: Yes!
Σ power(FACTOR, position)*isRelevant[User, searchResult[position].DocID]
Offline
Re-‐Query
Search
Engine
Query
Logs
New
Results
Big
Data
Array
Search
Engine
(possibly
embedded)](https://image.slidesharecdn.com/searchaccuracymetrics-paulnelson-151021174545-lva1-app6891/85/Search-Accuracy-Metrics-and-Predictive-Analytics-A-Big-Data-Use-Case-Presented-Paul-Nelson-Search-Technologies-12-320.jpg)
Search Accuracy Metrics and Predictive Analytics - A Big Data Use Case: Presented Paul Nelson, Search Technologies
- 1. O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
- 2. Search Accuracy Metrics & Predictive Analytics A Big Data Use Case Paul Nelson Chief Architect, Search Technologies pnelson@searchtechnologies.com
- 3. 3 There will be a demo (so don’t go away)
- 4. 4 185+ Consultants Worldwide San Diego London, UK San Jose, CR Cincinna> Prague, CZ Washington (HQ) Frankfurt, DE • Founded 2005 • Deep search expertise • 700+ customers worldwide • Consistent profitability • Search engines & Big Data • Vendor independent
- 5. 5 Typical Conversation with Customer Our search accuracy is bad How bad? Really, really, bad. Uh… on a scale of 1 to 10, how bad? An eight. No wait… a nine. Maybe even a 9.5. Let’s call it a 9.23
- 6. 6 Current methods are woefully inadequate • Golden Query Set o Key Documents • Top 100 / Top 1000 Queries Analysis • Zero result queries • Abandonment rate • Queries with click • Conversion
- 7. 7 What are we trying to achieve? • Reliable metrics for search accuracy • Can run analysis off-line o Does not require production deployment (!) • Can accurately compare two engines • Runs quickly = agility = high quality • Can handle different user types / personalization o Broad coverage • Provides lots of data to analyze what’s going on o Data to decide how best to improve the engine
- 8. Search Engine Under Evalua1on Search Engine Under Evalua1on Search Engine Under Evalua1on 8 Leverage logs for accuracy testing Query Logs Click Logs Big Data Framework • Engine Score(s) • Other metrics & histograms • Scoring database Search Engine Under Evalua1on
- 9. 9 From Queries à Users • User by User Metrics o Change in focus • Group activity by session and/or user o Call this an “Activity Set” o Merge sessions and users • Use Big Data to analyze all users o There are no stupid queries and no stupid users o Overall performance based on the experience of the users Queries Other Ac>vity Clicks Clusters User
- 10. 10 Engine Score • Group activity by session and/or user (Queries & Clicks) • Determine “relevant” documents o What did the user view? Add to cart? Purchase? o Did the search engine return what the user ultimately wanted? • Determine engine score per query based on user’s POV o Σ power(FACTOR, position)*isRelevant[user, searchResult[position].DocID] o (Note: many other formulae possible, MRR, MAP, DCG, etc.) • Average score for all user queries = user score • Average scores across all users = final engine score
- 12. 12 Off-Line Engine Analysis o Can we re-compute this array for all queries? o ANSWER: Yes! Σ power(FACTOR, position)*isRelevant[User, searchResult[position].DocID] Offline Re-‐Query Search Engine Query Logs New Results Big Data Array Search Engine (possibly embedded)
- 13. 13 Continuous Improvement Cycle Modify Engine Execute Queries Compute Engine Score Evaluate Results Log Files Search Engine Search Score Per Engine Version
- 14. 14 Watch the Score Improve Over Time
- 15. 15 What else can we do with Engine Scoring? Predictive Analytics
- 16. 16 The Brutal Truth about Search Engine Scores • Random ad-hoc formulae put together o No statistical or mathematical foundation • TF / IDF à All kinds of inappropriate biases o Bias towards document size (smaller / larger) o Bias towards rare (misspelled? archaic?) words o Not scalable (different scores on different shards) • Same formula since the 1970’s They are not based on science. We can do beKer!
- 17. Big Data Cluster 17 We use Big Data to Predict Relevancy Search Engine Content Sources Connectors Index Search Index Search Project Docs Web Site Pages Support Pages Landing Pages Content Processing Content Copy Search Click Logs Click Logs Query Logs Financial Data Business Data Query Logs Op Relevancy Model
- 18. 18 Probability Scoring / Predictive Relevancy clicked ? purchased ? 0 0 1 1 1 0 0 0 1 0 1 1 Predic1ve Analy1cs Sta1s1cal Model to Predict Probability Product Signals Query Signals User Signals Comparison Signals
- 19. 19 The Power of the Probability Score • The score predicts probability of relevancy • Value is 0 à 1 o Can be used for threshold processing o All documents too weak? Try something else! o Can combine results from different sources / constructions together • Identifies what’s important o Machine learning optimizes for parameters -‐ Identifies the impact and contribution of every parameter o If a parameter does not improve relevancy à REMOVE IT o Scoring becomes objective, not subjective (now based on SCIENCE) o Allows for experimentation on parameters
- 20. 20 And now the demo! (just like I promised)
- 21. Come out of the darkness
- 22. And into the Light!
- 23. The Age of Enlightenment for search engine accuracy is upon us!
- 24. Search Accuracy Metrics & Predictive Analytics A Big Data Use Case Paul Nelson Chief Architect, Search Technologies pnelson@searchtechnologies.com Thank you!