Making search better by tracking & utilizing user search behavior
2 likes3,572 views
The document discusses improving search functionalities through the analysis of user search behavior and metrics. It highlights the importance of measuring search quality using various metrics, including usage data and editorial labeling, to enhance result relevance. Additionally, it covers strategies for improving search results by leveraging user interactions and patterns to inform ranking systems.
Making search better by tracking & utilizing user search behavior
- 1. Making search better by tracking & utilizing user search behavior Sameer Maggon
- 2. Agenda • About Me ! • Measuring Search Quality ! • Quality Metrics based on users interactions ! • Quality Metrics based on User Tagging (labelers) ! • Improving Search Results ! • Using user behavior to improve results ranking - Example 2
- 3. Sameer Maggon Founder of Cloud based Zero Management Solution for Search Built relevance based search platform for AT&T Interactive & properties including yp.com, buzz.com, yellowpages.com Consulted for numerous Startups to Fortune 500 companies around Search & Discovery. 3 Engineering Alumni @maggon http://linkedin.com/in/maggon
- 4. Search seems Easy 1 2 3 Index all content Put a search box Show google 4 like results
- 5. Is It? • How do you know that your users are finding what they 5 are looking for? ! • How do you know what impact your one-off fix has on an aggregate? ! • Seemingly good result list to one might be irrelevant to another (e.g. mosaic)
- 6. How do we measure search? Usage Data Editorial Labeling 6 Collect, Analyze & Report on interactions users are having with your search functionality. Get a set of users to mark top x results with “Relevant vs. Not” for a pre-determined sample set of searches. ! Then compute specific metrics based on those.
- 7. Usage Data: Key Metrics to look at 7 • No Result Search % • Search Exits % • CTR % ! • Average Click Position • MRR (Mean Reciprocal Rank) • Clicks per Search • Paging (how deep do people have to dig?) ! • Latency (Average, tp90 and tp95) Aggregate & Trends Trends Aggregate & Trends
- 9. 9 Editorial Labeling • Precision Recall ! • DCG (Discounted Cumulative Gain) ! • nDCG (Normalized DCG) http://en.wikipedia.org/wiki/Discounted_cumulative_gain Relevant Not Relevant Relevant Not Relevant Not Relevant Relevant Not Relevant weight-age decreases as as you go down on an ordered list
- 10. Improving Search Results Attack Low Hanging fruit first • Popular No Result Searches - Can we use keyword stuffing? • Popular Search Exits - Eyeball outliers • Popular Searches with low CTRs • Generally improve Average Click Position / MRR via Utilizing Search Behavior to improve ranking • Utilize Popularity (click stream) to inform search ranking • Utilizing past search history to offer assistive features (search suggestions, related searches) 10 identifying patterns (impacts CTR) Advanced: Learning Models • Topic for some other time :)
- 11. Using Popularity to affect search ranking DEMO 11