Clarification Questions Usefulness (Slides)
0 likes31 views
The document investigates the role of clarifying questions (CQs) in enhancing conversational search experiences and user satisfaction. It explores features that contribute to the usefulness of CQs and employs machine learning classifiers to predict their effectiveness based on various attributes of search queries and CQs. Findings indicate that subjective and sentiment-driven CQs are more useful, and that query length impacts satisfaction levels.
Clarification Questions Usefulness (Slides)
- 1. Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness Hossein A. Rahmani1 , Xi Wang1 , Mohammad Aliannejadi2 , Mohammadmehdi Naghiaei3 , Emine Yilmaz1 1 University College London, London, UK {hossein.rahmani.22,xi-wang,emine.yilmaz}@ucl.ac.uk 2 University of Amsterdam, Amsterdam, The Netherlands m.aliannejadi@uva.nl 3 University of Southern California, California, USA naghiaei@usc.edu The 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL) Malta 17 to 22 of March, 2024
- 2. Asking Clarification Questions ● The pivotal role of Asking Clarifying Questions (CQs) in enhancing conversational search experiences ● Improving system performance and overall user experience using timely and high-quality CQs ● The importance of optimizing the formulation of CQs ● Investigating numerous features that contribute to the usefulness of clarifying questions ● Predicting the practical value – usefulness and user satisfaction – of CQs, based on various attributes of search queries, CQs, and their candidate answers
- 3. Experimental Setups ● Two commonly real-world datasets: ○ MIMICS and MIMICS-Duo ● Question-based features: (1) the question template variance, (2) clarifying question presentation with a varied number of candidate answers, (3) question subjectivity, (4) sentimental polarity of questions and (5) question length ● Query-oriented features: (6) query length in words, (7) query types (ambiguous or faceted) and (8) query-question relevance ● Investigating the value of the learned features on classifying the usefulness of clarifying questions
- 4. Experimental Setup (Cont.) ● Exploring various machine learning classifiers to estimate the usefulness of a given clarifying question ● Categorised the experimented approaches to: ○ Traditional Methods: Decision Tree Classifier (DTC), Random Forest Classifier (RFC) and Support Vector Classifier (SVC) with a linear kernel. ○ Neural-based Methods: BERT, DistilBERT, ALBERT, BART, GPT-4 ○ LLM-based Method: GPT-4 ● Evaluation metrics ○ Precision, Recall, and F1 score.
- 5. CQ Usefulness Question Templates Number of Candidate Answers
- 6. CQ Usefulness: Subjectivity and Sentiment ● Subjectivity refers to the degree to which a question expresses a belief rather than objective facts. ● In CQ, highly subjective questions may provide the desired level of clarification since they reflect the perspective of the questioner and may resonate with the user’s information needs. ● Sentiment polarity refers to the emotional tone of a question, typically measured as positive, negative, or neutral. ● Useful CQs are positively correlated with Subjectivity and Sentiment polarity.
- 7. CQ Usefulness: Question and Query Length ● As the query length increases, there is a noticeable decline in the rate of clarification usefulness. ● Users are more satisfied with short queries and long clarifying questions, suggesting that shorter queries can potentially lead to more ambiguity, creating room for the system to intervene.
- 8. CQ Usefulness: Ambiguous vs. Faceted Queries ● Clarifying questions for faceted queries are more useful than those for ambiguous queries. ● On MIMICS-Duo faceted queries have a better rate, ambiguous queries also receive a remarkable usefulness rate.
- 9. Clarifying Question Usefulness Prediction