Building Compassionate Conversational Systems
- 1. Building Compassionate and Personalized Conversational Systems - A point of view Presenter: Rama Akkiraju IBM Distinguished Engineer Acknowledgements: To our entire team in Watson
- 3. To build Compassionate and Personalized Conversational Systems, three core models are needed 3 Naturally3 (Mediums) Interact2 People1 1. Understand people at a deeper level 2. Understand styles of human interaction and optimize human-computer interaction 3. Understand and respond in various mediums in which interactions can occur • Need the ability to interact2 naturally3 with people1 Input Types: Text, Speech, Gestures Mediums: Computers, Mobile devices, Robots, Avatars
- 5. User Modeling: Our framework @Copyright IBM 2015 5 Act Be Feel Context Think Options Explore & Decide Inner State Environment Outer State An individual takes action based on the combination of his/her unique being & environment
- 6. User Modeling: Our framework @Copyright IBM 2015 6 Act Search Preferen ces Commun ications Decisions Commit ments Purchases Context Life Style, Events Sociological Economic Political Technological Options Price Promotions Products/ Services Place FeelPerceptions Emotions Sensations Attitudes Influences Sentiments Be Personality Needs, Values Beliefs Motives Identity Goals, Ambitions Interests Think Knowledge Skills Opinions Cognitive Style Explore & Decide Choices Consequenc es Session Intent Time
- 7. Use Personality Insights to engage with individuals at personalized level 7 Source: https://www.army.mil/article/78562/Leavi ng_the_battlefiel d__Soldi er_shares_story_of_PTSD https://watson-pi-demo.mybluemix.net/
- 11. Personalizing shopping Experience with Personality Insights 5/31/17Devoxx 201711
- 12. 5/31/17Page 12
- 13. Assessing Customer Satisfaction with Tones uuuu
- 15. Natural Interactions among People 15 Verbal (expressive, aggressive, passive) , Non-verbal (gestures, facial expressions, postures)
- 16. Dialog Act • Dialog Act is a specializedSpeech Act. Typically,looks at patterns in dialogs. 16 • Statement • backchannel/acknowledge • Opinion • abandoned/uninterpretable • agreement/accept • appreciation • yes-no-question • non-verbal • yes answers • conventional-closing • wh-question • no answers response • quotation • Summarize/reformulate • affirmative • action-directive • collaborative completion • repeat-phrase open-question • rhetorical-questions • reject • other answersconventional- opening or-clause • commits self-talk • downplayer • apology • thanking Source: Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech http://www.aclweb.org/anthology/J00-3003
- 17. Dialog Strategies Start Giving an extra Acknowledging Need Description of Need Anger Acknowledging w/out encouraging Refocus statements Active Listening Possibility of mistake Admitting mistake Allowing venting Apology Smiles Arranging Follow-up Need cannot be fulfilled on the spot Assurance of effort Assurance of result Mistake has been made Bonus buyoff Broken record Uncooperative customer Closing positively Common Courtesy Completing Follow-up Contact Security Aggressiveness Disengaging Distraction Frustration Empathy statement Expediting Expert Recommendati on Explain Reasoning or action Embarrassment Face-Saving Out Conflict Finding Agreement Points Following up Helpless Offering Choice Empowering Preventive strike Privacy insurance Privacy concern Probing question Pros and Cons Providing Alternatives Providing Takeaway Confusion Providing Explanation Questioning instead of stating Referral to supervisor Referral to 3rd party Lost focus Refocus Inappropriate behavior Setting Limits Critical Neutral mode Summarize the conversation Silence Thank-you Timeout Use customer name Verbal Softeners When QuestionYou re right Action Negative Emotion Monologue End External Giving Emotions General States Gratitude Statement Happiness Work by IBM Haifa Research team Michal Shmueli-Scheuer, Jonathan Herzig, Guy Feigenblat, David Konopnicki @Copyright IBM 2015
- 19. Mediums of interaction: On going work in Research • Text • Speech • Non-verbal clues: pauses, volume, intonation, pitch, • Video • Gestures, facial expressions, eye contact, posture, and tone of voice, distance, • Other • ? 19
- 20. Different channels for Conversations • Kiosks • Bots • Robots • Virtual agents on mobile-devices • Virtual agents accessible on a computer • Question from User modeling point of view. • Would user style of interaction with the system change based on devices/channels? • Would users willingness to reveal information about themselves change depending on the channel/device? 20
- 21. To build Compassionate and Personalized Conversational Systems, three core models are needed 21 Naturally3 (Mediums) Interact2 People1 1. Understand people at a deeper level 2. Understand styles of human interaction and optimize human-computer interaction 3. Understand and respond in various mediums in which interactions can occur • Need the ability to interact2 naturally3 with people1 Input Types: Text, Speech, Gestures Mediums: Computers, Mobile devices, Robots, Avatars
- 23. Tone Analyzer in Customer Support Q&A Forum Study #1: Clients’ Q&A forum data was analyzed • Confident responses are more likely to receive Kudos (r = 0.23) • Tentative responses are less likely to receive Kudos (r=0.27) • We found that we can predict kudos received with 66% accuracy which is better than random (50%) • We applied multiple state of the art classifiers such as Naïve Bayes, SVM, Random Forest and did 10-fold cross validation Study #2: Twitter customer support forums (333 conversations (240 Sat, 93 not-Sat)) • More angry customers are less likely to be satisfied after the conversation (r = -0.198) • More disgusted customers are less likely to be satisfied after the conversation (r = -0.184) • Agents who show higher emotional range are less likely to satisfy the customer (r = -0.186)
- 24. Personality Insights: Problem Setup • Given at least 1,500 words of text authored by an individual, infer the personality,needs and values of that individual. 24
- 25. Personality Insights Accuracy – Latest results 25 # of Tweets Mean Absolute Error (MAE) Trait Name Mean Absolute Error (MAE) Correlation Agreeableness 0.0999 0.2920 Conscientiousness 0.1174 0.3259 Extraversion 0.1477 0.2521 Neuroticism 0.1404 0.4182 Openness 0.0862 0.3650 • A Machine Learned model for predicting Personality Traits • UsesWord2Vec features (Stanford Glove pre-trainedmodel) • Ground truth collected include 2,000 psychometric surveys
- 26. How many words to infer Personality? 26 # of Tweets Mean Absolute Error (MAE) We reach 95% of the max accuracy with as low as 30 tweets. 0.09 0.095 0.1 0.105 0.11 0.115 0.12 0.125 0.13 0 50 100 150 200 250 300 350 MAE Number of tweets used for testing Trait Agreeableness – MAE VS numberof tweets Old Model New Model Old Model: Linguistic Inquiry Word Count (LIWC) based New Model: Word2Vec based
- 27. Greeting • Opening • Closing Statement • Give Info • Expressive (Pos/Neg) • Complaint • Offer Help • Suggest Action • Promise • Sarcasm • Other Request • Request Help • Request Info • Other Question • Yes-No Question • Wh- Question • Open Question Answer • Yes-Answer • No-Answer • Response-Ack • Other Social Act • Thanks • Apology • Downplayer Methodology • Designing more fine-grained actionable dialogue acts:
- 28. Data Collection • We gather annotations for 800 conversations (5,327 turns, ~6 turns/conversation on average, 4 different agent companies) using crowd workers. • They are asked to select as many categories as required to fully characterize the intent of the tweet.
- 29. 0 500 1000 1500 2000 2500 Full Data Distribution (@800 conversations, 5,327 turns)
- 30. Utterances are complex: A single label is not sufficient 0 50 100 150 200 250 300 350 400 450 500 (statement_info, answer_other) (statement_expressive_negative, statement_complaint) (statement_info, statement_complaint) (request_info, question_yesno) (request_info, question_wh) (request_info, question_open) (statement_offer, request_info) (statement_info, statement_expressive_negative) (request_info, socialact_apology) (statement_info, statement_suggestion) (statement_suggestion, request_info) (statement_info, socialact_thanks) (statement_info, answer_yes) (statement_info, request_info) (question_yesno, socialact_apology) (statement_info, question_yesno) § We test the hypothesis that each turn may require more than one dialogue act label by finding the distribution of label overlap in our annotations § We verify that labels frequently co-occur, so classification should assign an utterance multiple labels
- 31. Experimental Setup • We develop a sequential SVM-HMM model on the data • Labeling Modes: – Single label to a turn – Multiple labels to a turn • SVM-HMM learning methods: – Standard (future-looking HMM) – Online (model predicts a single label at a time, and cannot use future turns)
- 33. Class Division: 6, 8, and 10 (Easy & Hard) classes 33 6 Labels 8 Labels 10 Labels (Easy) 10 Labels (Hard) 1. Statement Informative 2.Request Information 3.Statement Complaint 4.Yes-No Question 5.Expressive Negative Statement 6. Other 1. Statement Informative 2.Request Information 3.Statement Complaint 4.Yes-No Question 5.Expressive Negative Statement 6.Statement Suggestion 7. GeneralAnswer 8. Other 1. Statement Informative 2.Request Information 3.Statement Complaint 4.Yes-No Question 5.Expressive Negative Statement 6.Statement Suggestion 7.GeneralAnswer 8.Apology Social Act 9.Thanking SocialAct 10.Other 1. Statement Informative 2.Request Information 3.Statement Complaint 4.Yes-No Question 5.Expressive Negative Statement 6.Statement Suggestion 7. GeneralAnswer 8. Statement Offer 9. Open Question 10. Other
- 34. SVM-HMM Sequential Model outperforms non- sequential baselines We expect a larger improvement by SVM-HMM with longer conversations (currently~6 turns/conversation)
- 35. Agents are more predictable than customers Prediction results are better when using *only* agent turns… Agent acts are less varied Customers are more difficult, but prediction is still good
- 36. Conversation outcomes are strongly distinguishable using predicted dialogue acts • Putting it all together: We ran outcome experiments using full conversation as input, and our predicted dialogue act labels as features • We balance the distribution of outcomes for each class: • Satisfied/not-satisfied (216 conversations/class) • Resolved/not-resolved (271 conversations/class) • Frustrated/not-frustrated (229 conversations/class) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Satisfaction Resolution Frustration LinSVC Dialogue Ngrams+HC Ngrams+HC+Dialogue Observations: • For satisfaction and resolution, dialogue act features are capturing all of the information in the n- grams, and they also are useful and explanatory • Frustration greatly benefits from handcrafted features – less accurately tied to just dialogue features.