Deep Learning for Dialogue Modeling - NTHU
Download as PPTX, PDF3 likes883 views
The document outlines the key components of a spoken dialogue system including natural language understanding, dialogue management, and output generation and discusses challenges such as predefined schemas and error propagation. It also describes recent trends in using deep learning techniques like recurrent neural networks for natural language understanding and end-to-end trainable dialogue systems. The document provides an overview of the architecture and progress of intelligent conversational agents.
Deep Learning for Dialogue Modeling - NTHU
- 2. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 2
- 3. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 3
- 4. Apple Siri (2011) Google Now (2012) Facebook M & Bot (2015) Intelligent Assistants 4 Google Home (2016) Microsoft Cortana (2014) Amazon Alexa/Echo (2014)
- 5. Why We Need? Daily Life Usage ◦ Weather ◦ Schedule ◦ Transportation ◦ Restaurant Seeking 5
- 6. Why We Need? – Get things done • E.g. set up alarm/reminder, take note – Easy access to structured data, services and apps • E.g. find docs/photos/restaurants – Assist your daily schedule and routine • E.g. commute alerts to/from work – Be more productive in managing your work and personal life 6
- 7. Why Natural Language? • Global Digital Statistics (2015 January) Global Population 7.21B Active Internet Users 3.01B Active Social Media Accounts 2.08B Active Unique Mobile Users 3.65B The more natural and convenient input of devices evolves towards speech. 7
- 8. Intelligent Assistant Architecture Reactive Assistance ASR, LU, Dialog, LG, TTS Proactive Assistance Inferences, User Modeling, Suggestions Data Back-end Data Bases, Services and Client Signals Device/Service End-points (Phone, PC, Xbox, Web Browser, Messaging Apps) User Experience “restaurant suggestions”“call taxi” 8
- 9. • Spoken dialogue systems are intelligent agents that are able to help users finish tasks more efficiently via spoken interactions. • Spoken dialogue systems are being incorporated into various devices (smart-phones, smart TVs, in-car navigating system, etc). Spoken Dialogue System (SDS) JARVIS – Iron Man’s Personal Assistant Baymax – Personal Healthcare Companion 9 Good dialogue systems assist users to access information conveniently and finish tasks efficiently.
- 10. APP BOT 10 Seamless and automatic information transferring across domains reduce duplicate information and interaction • A bot is responsible for a “single” domain, similar to an app 愛食記 地圖 LINE Goal: Schedule a lunch with Vivian KKBOX
- 11. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 11
- 12. System Framework 12 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • System Action/Policy Decision Output Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal current bottleneck error propagation
- 13. Interaction Example User Intelligent Agent Q: How does a dialogue system process this request? Good Taiwanese eating places include Din Tai Fung, Boiling Point, etc. What do you want to choose? I can help you go there. find a good eating place for taiwanese food 13
- 14. System Framework 14 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • System Action/Policy Decision Output Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal
- 15. 1. Domain Identification Requires Predefined Domain Ontology find a good eating place for taiwanese food User Organized Domain Knowledge (Database)Intelligent Agent 15 Restaurant DB Taxi DB Movie DB Classification!
- 16. 2. Intent Detection Requires Predefined Schema find a good eating place for taiwanese food User Intelligent Agent 16 Restaurant DB FIND_RESTAURANT FIND_PRICE FIND_TYPE : Classification!
- 17. 3. Slot Filling Requires Predefined Schema find a good eating place for taiwanese food User Intelligent Agent 17 Restaurant DB Restaurant Rating Type Rest 1 good Taiwanese Rest 2 bad Thai : : : FIND_RESTAURANT rating=“good” type=“taiwanese” SELECT restaurant { rest.rating=“good” rest.type=“taiwanese” }Semantic Frame Sequence Labeling O O B-rating O O O B-type O
- 18. System Framework 18 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • System Action/Policy Decision Output Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal
- 19. State Tracking Requires Hand-Crafted States User Intelligent Agent find a good eating place for taiwanese food 19 location rating type loc, rating rating, type loc, type all i want it near to my office NULL
- 20. State Tracking Requires Hand-Crafted States User Intelligent Agent find a good eating place for taiwanese food 20 location rating type loc, rating rating, type loc, type all i want it near to my office NULL
- 21. State Tracking Handling Errors and Confidence User Intelligent Agent find a good eating place for taixxxx food 21 FIND_RESTAURANT rating=“good” type=“taiwanese” FIND_RESTAURANT rating=“good” type=“thai” FIND_RESTAURANT rating=“good” location rating type loc, rating rating, type loc, type all NULL ? ?
- 22. Policy for Agent Action • Inform – “The nearest one is at Taipei 101” • Request – “Where is your home?” • Confirm – “Did you want Taiwanese food?” • Database Search • Task Completion / Information Display – ticket booked, weather information 22 Din Tai Fung : :
- 23. System Framework 23 Semantic Frame request_movie genre=action, date=this weekend Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • System Action/Policy Decision Hypothesis are there any action movies to see this weekend Text Input Are there any action movies to see this weekend? Speech Signal Output Generation System Action/Policy request_location Text response Where are you located? Screen Display location?
- 24. Output / NL Generation • Inform – “The nearest one is at Taipei 101” v.s. • Request – “Where is your home?” v.s. • Confirm – “Did you want Taiwanese food?” 24
- 25. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 25
- 26. Challenge • Predefined semantic schema Chen et al., “Matrix Factorization with Knowledge Graph Propagation for Unsupervised Spoken Language Understanding,” in ACL-IJCNLP, 2015. • Data without annotations Chen et al., “Zero-Shot Learning of Intent Embeddings for Expansion by Convolutional Deep Structured Semantic Models,” in ICASSP, 2016. • Semantic concept interpretation Chen et al., “Deriving Local Relational Surface Forms from Dependency-Based Entity Embeddings for Unsupervised Spoken Language Understanding,” in SLT, 2014. • Predefined dialogue states Chen, et al., “End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding,” in Interspeech, 2016. • Error propagation Hakkani-Tur et al., “Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM,” in Interspeech, 2016. • Cross-domain intention/bot hierarchy Sun et al., “An Intelligent Assistant for High-Level Task Understanding,” in IUI, 2016. Sun et al., “AppDialogue: Multi-App Dialogues for Intelligent Assistants,” in LREC, 2016. Chen et al., “Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken Language Understanding,” in ICMI, 2016. • Cross-domain information transferring Kim et al., “New Transfer Learning Techniques For Disparate Label Sets,” in ACL-IJCNLP, 2015. FIND_RESTAURANT rating=“good” rating=5? 4? HotelRest Flight Travel Trip Planning 26
- 27. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 27
- 28. A Single Neuron z 1w 2w Nw … 1x 2x Nx b z z zbias y z e z 1 1 Sigmoid function Activation function 1 w, b are the parameters of this neuron 28
- 29. A Single Neuron z 1w 2w Nw …1x 2x Nx b bias y 1 5.0"2" 5.0"2" ynot yis A single neuron can only handle binary classification 29 MN RRf :
- 30. A Layer of Neurons • Handwriting digit classification MN RRf : A layer of neurons can handle multiple possible output, and the result depends on the max one … 1x 2x Nx 1 1y … … “1” or not “2” or not “3” or not 2y 3y 10 neurons/10 classes Which one is max?
- 31. Deep Neural Network (DNN) • Fully connected feedforward network 1x 2x …… Layer 1 …… 1y 2y …… Layer 2 …… Layer L …… …… …… Input Output MyNx vector x vector y Deep NN: multiple hidden layers MN RRf :
- 32. RNN for SLU • IOB Sequence Labeling for Slot Filling • Intent Classification 32 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 𝑓 ℎ1 𝑓 ℎ2 𝑓 ℎ 𝑛 𝑓 ℎ0 𝑏 ℎ1 𝑏 ℎ2 𝑏 ℎ 𝑛 𝑏 𝑦0 𝑦1 𝑦2 𝑦 𝑛 (a) LSTM (b) LSTM-LA (c) bLSTM-LA (d) Intent LSTM intent 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛 𝑦0 𝑦1 𝑦2 𝑦 𝑛 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛 𝑦0 𝑦1 𝑦2 𝑦 𝑛 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛
- 33. RNN for SLU • Joint Multi-Domain Intent Prediction and Slot Filling – Information can mutually enhanced 33 semantic frame sequence ht-1 ht+1ht W W W W taiwanese B-type U food U please U V O V O V hT+1 EOS U FIND_REST V Slot Tagging Intent Prediction Hakkani-Tur, et al., “Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM,” in Interspeech, 2016.
- 34. 34 just sent email to bob about fishing this weekend O O O O B-contact_name O B-subject I-subject I-subject U S I send_email D communication send_email(contact_name=“bob”, subject=“fishing this weekend”) are we going to fish this weekend U1 S2 send_email(message=“are we going to fish this weekend”) send email to bob U2 send_email(contact_name=“bob”) B-message I-message I-message I-message I-message I-message I-message B-contact_nameS1 Domain Identification Intent Prediction Slot Filling Contextual SLU (Chen et al., 2016)
- 35. 35 u Knowledge Attention Distributionpi mi Memory Representation Weighted Sum h ∑ Wkg o Knowledge Encoding Representation history utterances {xi} current utterance c Inner Product Sentence Encoder RNNin x1 x2 xi… Contextual Sentence Encoder x1 x2 xi… RNNmem slot tagging sequence y ht-1 ht V V W W W wt-1 wt yt-1 yt U U RNN Tagger M M Chen, et al., “End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding,” in Interspeech, 2016. 1. Sentence Encoding 2. Knowledge Attention 3. Knowledge Encoding Contextual SLU (Chen et al., 2016) Idea: additionally incorporating contextual knowledge during slot tagging track dialogue states in a latent way
- 36. E2E Supervised Dialogue System 36Wen, et al., “A Network-based End-to-End Trainable Task-Oriented Dialogue System,” arXiv.:1604.04562v2. 0 0 0 … 0 1 Database Operator Copy field … Database Sevendays CurryPrince Nirala RoyalStandard LittleSeuol DB pointer Can I have korean Korean 0.7 British 0.2 French 0.1 … Belief Tracker Intent Network Can I have <v.food> Generation Network <v.name> serves great <v.food> . Policy Network zt pt xt MySQL query: “Select * where food=Korean” qt
- 37. InfoBot: E2E Dialogue System with Supervised & Reinforcement Learning 37Dhingra, et al., “End-to-End Reinforcement Learning of Dialogue Agents for Information Access,” arXiv.:1609.00777v2. Movie=?; Actor=Bill Murray; Release Year=1993 Find me the Bill Murray’s movie. I think it came out in 1993. When was it released? Groundhog Day is a Bill Murray movie which came out in 1993. KB-InfoBot User (Groundhog Day, actor, Bill Murray) (Groundhog Day, release year, 1993) (Australia, actor, Nicole Kidman) (Mad Max: Fury Road, release year, 2015) Knowledge Base (head, relation, tail) Idea: differentiable database for propagating the gradients
- 38. TC-Bot: E2End Task-Completion Bot 38 wi B- type wi+ 1 wi+2 O O EOS <intent> wi B- type wi+ 1 wi+2 O O EOS <intent> Semantic Frame request_movie genre=action, date=this weekend System Action / Policy request_location User Dialogue Action Inform(location=San Francisco) Time t-1 wi <slot> wi+ 1 wi+2 O O EOS <intent> Language Understanding (LU) Time t-2 Time t Dialogue Management (DM) w0 w1 w2 Natural Language Generation (NLG) EOS User Goal User Agenda Modeling User Simulator End-to-End Neural Dialogue System Text Input Are there any action movies to see this weekend? Idea: supervised learning for each component and reinforcement learning for end-to-end training the neural dialogue system Li, et al., “End-to-End Task-Completion Neural Dialogue Systems,” arXiv.:1703.01008.
- 39. RL TC-Bot User goal: Two tickets for “the witch” tomorrow 9:30 PM at regal meridian 16, Seattle. 39 RULE BASED SYSTEM User: Can I get 2 tickets for the witch? Agent: Which theater do you want? User: Regal meridian 16 Agent: What time would you like to see it? User: 9:30 pm Agent: What date would you like to see it? User: Tomorrow! Agent: How many people are you? User: Just two adults. Agent: What is your location? User: Seattle! Agent: What kind of video format do you like? User: Oh, any kind is okay. Agent: Do you have any price constraint? User: No, I don’t care. Agent: Great - I was able to purchase 2 tickets for you to see the witch tomorrow at regal meridian 16 theater in Seattle at 9:30 pm. User: Thank you. Agent: Thank you! REINFORCEMENT LEARNING SYSTEM User: Can I get 2 tickets for the witch? Agent: What date would you like to see it? User: Tomorrow Agent: Which theater do you prefer? User: Regal meridian 16 Agent: What is your location? User: Seattle Agent: What time would you like to see it? User: 9:30 pm. Agent: Great - I was able to purchase 2 tickets for you to see the witch tomorrow at regal meridian 16 theater in Seattle at 9:30 pm. User: Thanks. Agent: Thanks! The system can learn how to efficiently interact with users for task completion
- 40. Outline Introduction Framework Natural/Spoken Language Understanding Dialogue Management Output Generation Challenge Recent Trend Conclusion 40
- 41. Conclusion • The conversational systems can manage information access via spoken interactions • A domain is usually constrained by the backend service – Semantic schema should be predefined – Cross-domain knowledge and intention is difficult to handled • NN-Based Dialogue System – Pipeline outputs are represented as vectors distributional • Semantic frames as vectors to encode confidence • Implicitly represent dialogue states in hidden vectors – The execution is constrained by backend services symbolic 41
- 42. Q & A T H A N K S F O R YO U R AT T E N T I O N ! 42