Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
- 1. Asking Clarifying Questions in Open-Domain Information- Seeking Conversations Mohammad Aliannejadi(1), Hamed Zamani(2), Fabio Crestani(1), and W. Bruce Croft(2) (1) Università della Svizzera italiana (USI), Switzerland (2) University of Massachusetts Amherst, USA
- 5. 5
- 6. 6
- 7. 7
- 8. 8
- 12. Can we ask questions to clarify the user information needs? Johannes Kiesel et al. Toward Voice Query Clarification. SIGIR 2018 Radlinski and Craswell. A Theoretical Framework for Conversational Search. CHIIR 2017
- 15. How to evaluate?
- 16. ClueWeb Collection • A part of the Lemur Project • A common web crawl (English) with 50M documents • TREC Web Track 2009 – 2012 • Ad-hoc retrieval and diversification tasks
- 17. TREC facets
- 18. An offline evaluation methodology • We assume that each user is interested in one facet per topic.
- 19. An offline evaluation methodology • Let be the set of topics (queries). • A collection of facet sets: • includes all defined facets for topic . • A collection of clarifying question sets: • With including all clarifying questions relevant to topic . • An offline evaluation requires defining . • Borrowed from the ClueWeb Collection
- 23. Question Verification and Facet Linking • Two main concerns: • Precision: how is the quality of the collected clarifying questions? • Recall: are all facets addressed by at least one clarifying question? • Two expert annotators: • Marked invalid and duplicate questions. • Linked questions to the facets they found relevant. • Facets with no questions: generated new questions relevant to them.
- 25. An offline evaluation methodology • Let be the set of topics (queries). • A collection of facet sets: • includes all defined facets for topic . • A collection of clarifying question sets: • With including all clarifying questions relevant to topic . • An offline evaluation requires defining . • Borrowed from the ClueWeb Collection
- 27. Quality Check • Regular quality checks on the collected answers. • Manual checks on 10% of submissions per worker. • If any invalid answer was observed, we then checked all the submissions of the corresponding worker. • Invalid answers were removed and workers banned from future tasks. • Disabled copy/paste feature. • Monitored keystrokes.
- 28. Qulac: Questions for Lack of Clarity Qulac has two meanings in Persian: • blizzard • wonderful or masterpiece © HBO
- 29. Learning to ask clarifying questions
- 31. Question Retrieval • Task: Given a topic and a context (question-answer history), retrieve clarifying questions. • Desired objective: high recall • Approaches: • Term matching retrieval models: language models, BM25, RM3 (query expansion) • Learning to rank: LambdaMART, RankNet, neural ranking models (e.g., BERT)
- 34. Question Selection • Task: selecting a clarifying question that leads to retrieval improvement • Objective: high precision (in retrieval) • Approaches: • Query performance prediction (QPP): predicting the retrieval performance after asking each question (without answer) and selecting the one with the highest QPP. • Learning to rank: defining a set of features for ranking questions. The features include QPP, similarity to the topic, similarity to the context, etc. • Neural ranking models: learning to rank with representation learning (e.g., BERT)
- 35. Question Selection Asking only one good question improves the performance by over 100%.
- 40. Case Study Negative answer; new information. Retrieval model fails.
- 41. Case Study Open question; new information.
- 42. Future Directions • Utilizing positive and negative feedback for document retrieval. • Joint modeling of question retrieval and selection. • Question generation. • Determining the number of questions to ask based on the system’s confidence. • Explore other ways of evaluating a system: • Conversation turns; • Retrieval performance.
- 43. Conclusions • Asking clarifying questions in open-domain information-seeking conversations. • Qulac: a collection for automatic offline evaluation of asking clarifying questions for conversational IR. • A simple yet effective retrieval framework. • Asking only one good question improves the performance by over 100%! • More improvement for: • Shorter queries; • Ambiguous queries.
- 44. Questions? Qulac is publicly available at http://bit.ly/QulacData Thanks to SIGIR for the student travel grant!
Editor's Notes
- #5: <number>
- #13: make a frame of slides with logos... <number>
- #14: make it more dialogue <number>
- #15: put confidence here in the image <number>
- #33: change this <number>
- #43: remove one or two <number>
- #44: more stress on confidence merge with future directions <number>