Why User Behavior Insights? KMWorld Enterprise Search & Discovery 2024
0 likes17 views
The document discusses the importance of user behavior insights in improving search functionality, noting that traditional search often fails to meet user needs due to its diverse applications and user requirements. It emphasizes the necessity of collecting and analyzing behavioral data to optimize search performance and recommends various testing and tuning techniques. Furthermore, it outlines the significance of a hybrid search configuration that effectively integrates multiple search methodologies based on user interactions.
Why User Behavior Insights? KMWorld Enterprise Search & Discovery 2024
- 1. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Why User Behavior Insights? KMWorld Enterprise Search November 21, 2024 S T A V R O S M A C R A K I S , P R O D U C T M A N A G E R , O P E N S E A R C H @ A W S E R I C P U G H C O - F O U N D E R , O P E N S O U R C E C O N N E C T I O N S
- 2. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 2 Search is strategic
- 3. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 3 Search is strategic
- 4. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 4 And yet it never seems to work right
- 5. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 5 And yet it never seems to work right
- 6. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 6 Why is it hard? And yet it never seems to work right Search is strategic
- 7. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. mbedded • Company search within • Person search within email • Applications • Document search and recommendations (heavy on text) • Academic documents for PhD researchers • News for general public, for journalist • Intranet, highly heterogenous, inconsistent formats • Knowledge management for researchers, for sales people, for application engineers • Legal and regulatory, both high-precision (relevant regulation) and high- recall (discovery) • FAQs, call centers, troubleshooting • Looking for items, not documents (heavy on structured data) • E-commerce search and recommendations: looking for a thing to buy • Anything from plumbing parts to real estate to yoga lessons • Media – movies, TV, … • Job search; placements xpert search for interns, for research agents, for hedge fund managers urant, flight search, fractional jets – real-time availability e position ilarity; SmugMug very different from varied use cases varied datasets varied users 7 one size DoEsn’t fit aLl
- 8. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 8 How can you fix it?
- 9. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Conference survey • Understand application data • Talk to users and sponsors • Regression testing • Performance testing • Log search actions • Multilingual coverage • Evaluate quality of results • Privacy 9 What search needs
- 10. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. What search needs 10 Conference survey • Talk to users and sponsors • Privacy • Understand application data • Evaluate quality of results • Log search actions • Performance testing • Regression testing • Multilingual coverage
- 11. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Sort by gaps 11 Conference survey • Performance testing • Multilingual coverage • Regression testing • Evaluate quality of results • Talk to users and sponsors • Privacy • Understand application data • Log search actions
- 12. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. What most needs improvement 12 Conference survey • Log search actions • Understand application data • Evaluate quality of results • Talk to users and sponsors • Regression testing • Performance testing • Multilingual coverage • Privacy
- 13. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. What are even big users missing? 13 Conference survey • Log search actions • Understand application data • Evaluate quality of results • Talk to users and sponsors • Regression testing • Performance testing • Multilingual coverage • Privacy AWS customer gaps • Search specialist engineers • Search is a discipline • Logging • Quality evaluation • Incorporating multiple signals • Tuning
- 14. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 14 Measurement and analysis • Behavioral data • Online evaluation • Offline evaluation • A/B testing Search processing • Lexical search • Semantic search • Sparse neural Search tuning • Manual tuning • Hybrid search • Semantic reranking • Learning to Rank (LTR) The virtuous circle of search improvement
- 15. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. § Query understanding § Query rewriting § Lexical search § Neural sparse retrieval § Semantic vector search § Multimodal search § LLM reranking § LLM summarization How do you choose the best technique(s)?
- 16. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Tune 16 Measurement and analysis • Search behavior • Online evaluation tools • A/B testing • Creation of judgement sets • Offline evaluation tools • Metrics Search processing • Lexical search • ML: Semantic search • ML: Query understanding • ML: Sparse neural • ML: Relevance reranking • ML: Multimodal Tuning • Hybridizing • Regression evaluation • Manual tuning of signals • ML: Bayesian optimization (LTB) • ML: Semantic model fine-tuning • ML: Learning to Rank (LTR)
- 17. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Combined strengths 17 Semantic search Lexical search Hybrid search
- 18. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. So how do you tune and combine them? § Query understanding + § Query rewriting + § Lexical search + § Neural sparse retrieval + § Semantic vector search + § Multimodal search + § LLM reranking + § LLM summarization Hybrids Tuning (field boosts, …) Retrieval Augmented Generation (RAG) +
- 19. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. You need to collect and analyze data 19 Measurement and analysis • Search behavior • Online evaluation tools • A/B testing • Creation of judgement sets • Offline evaluation tools • Metrics Search processing • Lexical search • ML: Semantic search • ML: Query understanding • ML: Sparse neural • ML: Relevance reranking • ML: Multimodal Search tuning • Hybridizing • Regression evaluation • Manual tuning of signals • ML: Bayesian optimization (LTB) • ML: Semantic model fine-tuning • ML: Learning to Rank (LTR)
- 20. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 20 User Behavior Search Results Conversion
- 21. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 21 User Behavior – clicks Search Results Conversion 14:23 SEARCH_CLICK SearchID: 900354 UserID:969 Query: good to be the king louis 14:25 PLAY_CLICK SearchID: 900254 UserID:969 ObjectID: 9393 14:24 RESULT_CLICK SearchID: 900254 UserID:969 ObjectID: 9393 Position: 4 14:23 RESULTS SearchID: 900354 UserID:969 Object 2323, 5498, 3434, 9393, 539
- 22. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. User Behavior – end-to-end record 22 User 969 at 14:23 Search 900254 for “good to be the king louis” Results Object 2323, 5498, 3434, 9393, 5394, 9 Choose Result 4 = Object 9393 Play Object 9393 = UPC 490404333
- 23. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. DEMO 23
- 24. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Build !
- 25. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Build !
- 26. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. Metrics Data structure designed for ease of: aggregation, filtering and grouping Covers multiple use cases: Quick summaries of performance using different metrics Investigate performance regressions over time Compare search configurations Evaluate performance under different query sets (main sample, top queries, etc) Deep dive into best and worst performing queries Search Quality Metrics
- 27. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.
- 28. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.
- 29. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 29 Example of relevance comparison
- 30. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. 30 4. Deploy Best Hybrid Search Configuration 3. Run Hybrid Search Optimizer 2. Calculate Implicit Judgements 1. Collect User Signals (=UBI Data) Hybrid Search Optimizer Based on User Behavior Insights Data Start over again! q Interleaved A/B tests q Bandit algorithms q Establishing guardrails Hybrid search optimizer…
- 31. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. User Behavior Insights • Collect fine-grained behavioral data • Standard schema for search, search results, actions on search results • Trace actions on search results causally • Feed both manual and ML tuning
- 32. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark. https://www.ubisearch.dev/