End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
9 likes1,524 views
This document summarizes a research paper on end-to-end joint learning of natural language understanding and dialogue management. The paper proposes an end-to-end deep hierarchical model that leverages multi-task learning using three supervised tasks: user intent classification, slot tagging, and system action prediction. The model outperforms previous pipelined models by accessing contextual dialogue history and allowing the dialogue management signals to refine the natural language understanding through backpropagation. Evaluation on a dialogue state tracking dataset shows the joint model achieves better dialogue management performance compared to baselines and also improves natural language understanding.
![What is a Dialogue System?
2
Apple Siri (2011)
Facebook M & Bot (2015) Google Home (2016)
Microsoft Cortana (2014) Amazon Alexa/Echo (2014)
Category:
• Chit Chat
• Task-Oriented
[1 ] Zue, Victor W., and James R. Glass. “Conversational interfaces: Advances and challenges.” Proceedings of the IEEE 88.8 (2000): 1166-1180.
A dialogue system is a computer agent that interacts with human via natural languages.](https://image.slidesharecdn.com/icassp17e2eslide-170310225457/85/End-to-End-Joint-Learning-of-Natural-Language-Understanding-and-Dialogue-Manager-2-320.jpg)
![Appendix - Baseline Model
NLU: CRF for slot tagging, and
One-Vs-All SVMs for intent classification
SAP: One-Vs-All SVMs
[1] Raymond, Christian, and Giuseppe Riccardi. “Generative and discriminative algorithms for spoken language
understanding." In INTERSPEECH, pp. 1605-1608. 2007.
Predicting system actions at the next
turn as responses to the current user
behaviors by pipelining NLU and SAP
together
15](https://image.slidesharecdn.com/icassp17e2eslide-170310225457/85/End-to-End-Joint-Learning-of-Natural-Language-Understanding-and-Dialogue-Manager-15-320.jpg)
End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
- 1. xyang45@illinois.edu y.v.chen@ieee.org End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager ICASSP 2017 March 9th XUESONG YANG YUN-NUNG (VIVIAN) CHEN DILEK HAKKANI-TÜR HLT-L2: Spoken Language Understanding I PAUL CROOK XIUJUN LI JIANFENG GAO LI DENG
- 2. What is a Dialogue System? 2 Apple Siri (2011) Facebook M & Bot (2015) Google Home (2016) Microsoft Cortana (2014) Amazon Alexa/Echo (2014) Category: • Chit Chat • Task-Oriented [1 ] Zue, Victor W., and James R. Glass. “Conversational interfaces: Advances and challenges.” Proceedings of the IEEE 88.8 (2000): 1166-1180. A dialogue system is a computer agent that interacts with human via natural languages.
- 3. Pipelined Task-Oriented Dialogue System Automatic Speech Recognition Natural Language Understanding (NLU) • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking (DST) • Dialogue Policy Decision Natural Language Generation (NLG) Hypothesis Any action movies recommended this weekend? Semantic Frame request_movie (genre=action, date=this weekend) System Action/Policy request_location Text response Which theater do you prefer? Text Input Any action movies recommended this weekend? Speech Signal Motivation: The pipelined system (NLU DM) results in error propagation issues. 3 Text-To-Speech
- 4. Proposed Approach End-to-end model Mitigate the effects of noisy output from NLU Refine NLU by supervised signals from DM Multi-task jointly learning NLU - User intent classification NLU - User slot tagging DM - System action prediction Contextual understanding Access to the user history Monitor user behavior states over turns 4
- 5. Human-Human Dialogue Interaction Hi, how may I help you? Do you want to have a backpack type of hotel? So you don’t mind if it is not roomy, right? Okay. These hotels are available for you: … Thanks, goodbye. Are there any cheap rate hotels to put my bags? Yes. Just gonna leave our things there and stay out the whole day. Yes. Ok, thank you, bye! Idea: predicting the next system action given the current user utterance together with the aggregated observations Guide Agent Tourist User 5
- 6. Natural Language Understanding (NLU) 6 Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 1: Slot Tagging Task 2: Multi-Label User Intent Prediction Sharedweights
- 7. Task 1+2: Natural Language Understanding NLU+DM 1: Pipelined BLSTMs 7 single user turn current user turn w/ contextual history Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 3: Multi-Label System Action Prediction
- 8. NLU+DM 2: End-to-End Model (JointModel) 8 Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 1+2+3: End-to-End Joint NLU+DM supervised learning with three tasks DM output signal refines NLU for better robustness DM
- 9. Data Dialogue State Tracking Challenge 4 Human-human dialogues: 21-hour dialogue sessions on touristic information collected via Skype between tour guides and tourists User Intent = Speech Act + Attributes System Action = Speech Act + Attributes Domains Accommodation Attraction Food Shopping Transportation Speech Act QST (QUESTION) RES (RESPONSE) INI (INITIATIVE) FOL (FOLLOW) Speech Act Attributes ACK CLOSING COMMIT THANK CANCEL CONFIRM ENOUGH WHAT EXPLAIN HOW_MUCH HOW_TO WHEN INFO NEGATIVE OPENING WHERE POSITIVE PERFERENCE RECOMMEND WHO Train Dev Test #utter 5,648 1,939 3,178 #intent 68 54 58 9
- 10. DM Result – System Action Prediction (SAP) 10 Metric: frame-level accuracy (FrmAcc) Model FrmAcc Baseline (CRF+SVMs) 7.7 Pipeline-BLSTM 12.0 JointModel 22.8 Pipeline-BLSTM and JointModel outperform the baseline JointModel improves Pipeline-BLSTM about 10% accuracy, indicating the importance of mitigating downside of pipeline Human-human conversations are complicated, so predicting system actions for DM is difficult
- 11. DM Result – System Action Prediction (SAP) 11 Metric: frame-level accuracy (FrmAcc) Model FrmAcc Baseline (CRF+SVMs) 7.7 Pipeline-BLSTM 12.0 JointModel 22.8 Oracle-SAP (SVM) 7.7 Oracel-SAP (BLSTM) 19.7 Oracle models show the upper-bound of the SAP performance, since it transfers the errors from NLU to SAP Contextual user turns make significant contribution to DM performance JointModel achieves the best DM performance (FrmAcc) with richer latent representations Fully correct NLU output
- 12. NLU Result – Slot Filling & Intent Prediction 12 Metrics: frame-level accuracy (FrmAcc) Models Slot Intent NLU NLU-Baseline (CRF+SVM) 77.3 37.2 33.1 NLU-Pipeline-BLSTM 76.8 40.0 36.4 NLU-JointModel 76.5 42.2 37.4 CRF+SVMs baseline maintains strong NLU performance with 33.1% Pipeline-BLSTM and JointModel outperformed the baseline DM signal (system action prediction) helps more on user intent prediction than slot filling, and NLU is significantly improved Extra supervised DM signal helps refine the NLU by back-propagating the associated errors
- 13. Conclusion 13 First propose an end-to-end deep hierarchical model for joint NLU and DM with limited contextual dialogue memory Leverage multi-task learning using three supervised signals NLU: User intent classification NLU: Slot tagging DM: System action prediction Outperform the state-of-the-art pipelined NLU and DM models Better DM due to the contextual dialogue memory Robust NLU fine-tuned by supervised signal from DM
- 14. Code Available at https://github.com/XuesongYang/end2end_dialog Thanks for Your Attention! 14
- 15. Appendix - Baseline Model NLU: CRF for slot tagging, and One-Vs-All SVMs for intent classification SAP: One-Vs-All SVMs [1] Raymond, Christian, and Giuseppe Riccardi. “Generative and discriminative algorithms for spoken language understanding." In INTERSPEECH, pp. 1605-1608. 2007. Predicting system actions at the next turn as responses to the current user behaviors by pipelining NLU and SAP together 15
- 16. Appendix: Configuration Optimizer: a mini-batch stochastic gradient descent method Adam Contextual history: five user turns Dimension of word embedding: 512 Dropout ratio: 0.5 No early stopping, but use 300 training epochs Best models for three tasks are selected individually under different metrics Token-level micro-average F1 score is used for slot filling frame-level accuracy (it counts only when the whole frame parse is correct) is used for both user intent prediction and system action prediction the decision thresholds are tuned on dev set 16