Dstc6 an introduction
1 like185 views
This document summarizes the three tracks of the DSTC6 dialogue system technology challenges. Track 1 focuses on end-to-end goal oriented dialog learning with a restaurant reservation domain. Track 2 focuses on end-to-end conversation modeling to generate responses using dialogue history and external knowledge. Track 3 aims to detect breakdowns in dialogues using classification metrics.
Dstc6 an introduction
- 1. DSTC6 – Dialogue System Technology Challenges An Introduction 서강대학교 자연어처리 연구실 허광호 2017.07.12
- 2. DSTC6 Tracks • Track 1 – End-to-End Goal Oriented Dialog Learning • Y-Lan Boureau et al. - Facebook AI Research • Track 2 – End-to-End Conversation Modeling • Chiori HORI et al. (Mitsubishi Electric Research Laboratories) • Track 3 – Dialog Breakdown Detection • Ryuichiro Higashinaka et al. (NTT)
- 3. Track 3 – Dialog Breakdown Detection NB: Not a breakdown PB: Possible breakdown B: Breakdown
- 4. Track 3 – Dialog Breakdown Detection • 필요한 이유 • Voice agent 서비스가 상업용으로 출시되고 있지만 • Still cannot converse as naturally as two humans. • 가장 큰 문제점은 Voice agent가 가끔 Dialogue breakdown을 유발하는 부적절한 발화를 생성함. • 용도 • Breakdown detection 기술은 Chat-oriented 대화와 같이 대화유지가 중요한 경우 유용함. • 대화 시스템의 error recovery에도 사용할 수 있음.
- 5. Track 3 – Dialog Breakdown Detection • Dataset • 100 chat-oriented dialogues (21 utterances per dialogue) – 24 annotators. • 1000 chat-oriented dialogues – 2~3 annotators. • 300 chat-oriented dialogues – 30 annotators. • Unfortunately, the data above are in Japanese; • 추가로 영어로 된 100 dialogues 를 수집하여 배포한다고 함. • 평가방법 • Classification-Related metrics – Accuracy, Precision, Recall, F-measure • Distribution-related metrics – JS Divergence and Mean squared error
- 6. Track 3 – Dialog Breakdown Detection • LREC 2016 Breakdown Detection (In Japanese) 결과 • Baseline: CRF-based method • Team1: LSTM-RNN-based method • Features: Word2Vec + co-occurrence freq. vector + Sent2Vec vector • Team2: LSTM-RNN-based method (Word2Vec) • Team3: Rule-based method (Keyword는 시스템 발화에서 추출) • Team4: SVM-based method (Word frequency vector) • Team5: DNN-based method • Features: dialogue act of the system and previous user utterance. • Team6: LSTM-RNN-based method • Features: Word vector encoded by the use of NCM (Neural Conversation Model), LSTM, bag-of-word embedding, and an extended NCM.
- 7. Track 3 – Dialog Breakdown Detection Classification-Related Metrics Baseline: CRF Team1: LSTM-RNN-based Team2: LSTM-RNN-based Team3: Rule-based Team4: SVM-based Team5: DNN-based Team6: LSTM-RNN based 출처: The Dialogue Breakdown Detection Challenge - Task Description, Datasets, and Evaluation Metrics
- 8. Track 3 – Dialog Breakdown Detection Distribution-Related Metrics Baseline: CRF Team1: LSTM-RNN-based Team2: LSTM-RNN-based Team3: Rule-based Team4: SVM-based Team5: DNN-based Team6: LSTM-RNN based 출처: The Dialogue Breakdown Detection Challenge - Task Description, Datasets, and Evaluation Metrics
- 9. Track 1 – End-to-End Goal Oriented Dialog Learning • Goal-oriented Dialog Learning • Goal-oriented 대화는 language modeling 이상의 기술을 필요로 함. • Asking questions to clearly define a user request. • Querying Knowledge Bases (KBs). • Interpreting results from queries to display options to users or Completing a transaction. • 대화 도메인 • Restaurant reservation system • Facebook AI Research open resource 를 코퍼스로 사용 (Bordes et al. 2017)
- 10. Track 1 – End-to-End Goal Oriented Dialog Learning • Task 구성 • Goal-oriented 대화시스템이 갖춰야 할 기능들을 sub-task로 나누어서 각각 평가. • Task 1: Issuing API calls • Task 2: Updating API calls • Task 3: Displaying options • Task 4: Providing extra information • Task 5: Conducting full dialogs
- 11. Track 1 Example
- 12. Track 1 – End-to-End Goal Oriented Dialog Learning • Dataset 구성 • Task 당 10,000 examples • 정답 발화 + 발화 candidates • Evaluation • Language generation 방식이 아니라 • 발화 candidates Ranking 방식으로 진행됨 • Next-Utterance Classification 이라고 함 • (Lowe et al. 2016)
- 13. Track 1 – End-to-End Goal Oriented Dialog Learning Results published in ICLR 2017 출처: learning end-to-end goal-oriented dialog – A. Bordes 2017
- 14. Track 2 – End-to-End Conversation Modeling • The system has to generate sentences responsive to a user input in a given dialogue history where it can use external knowledge from web.
- 15. Track 2 – End-to-End Conversation Modeling • Dataset • Training Data (OpenSubtitles/Twitter ≈ 1M dialogs, 2.2M utterances) • Test Data: 500 – 1000 dialogs
- 16. Track 2 – End-to-End Conversation Modeling • Baseline System • LSTM-based seq2seq generation system and a pre-trained model will be provided. • Evaluation • Objective measure: Perplexity, BLEU, etc. • Subjective measure: Human rating using crowd source.
- 17. Track 2 – End-to-End Conversation Modeling
- 18. DSTC6 Tracks Conclusion • Track 1 – End-to-End Goal Oriented Dialog Learning • Next-Utterance Classification Task • Track 2 – End-to-End Conversation Modeling • Language Generation Task • Track 3 – Dialog Breakdown Detection • Label Classification Task