Deep Learning Models for Question Answering
Download as PPTX, PDF30 likes14,530 views
This document discusses deep learning models for question answering. It provides an overview of common deep learning building blocks such as fully connected networks, word embeddings, convolutional neural networks and recurrent neural networks. It then summarizes the authors' experiments using these techniques on benchmark question answering datasets like bAbI and a Kaggle science question dataset. Their best model achieved an accuracy of 76.27% by incorporating custom word embeddings trained on external knowledge sources. The authors discuss future work including trying additional models and deploying the trained systems.
Deep Learning Models for Question Answering
- 1. DEEP LEARNING MODELS FOR QUESTION ANSWERING Sujit Pal & Abhishek Sharma Elsevier Search Guild Question Answering Workshop October 5-6, 2016
- 2. About Us Sujit Pal Technology Research Director Elsevier Labs Abhishek Sharma Organizer, DLE Meetup and Software Engineer, Salesforce 2
- 3. • Watching Udacity “Deep Learning” videos taught by Vincent Vanhoucke. • Watching “Deep Learning for Natural Language Processing” (CS 224d) videos taught by Richard Socher. • Thinking that it might be interesting to do “something” around Question Answering and Deep Learning. How we started 3
- 8. Research 8
- 9. This had just ended… and the #4 ranked entry used Deep Learning for their solution. 9
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. 14
- 15. 15
- 17. Most DL Networks (including Question Answering models) composed out of these basic building blocks. • Fully Connected Network • Word Embedding • Convolutional Neural Network • Recurrent Neural Network Building blocks 17
- 18. Fully Connected Network Credit: neuralnetworksanddeeplearning.com • Workhorse architecture of Deep Learning. • Number of layers and number of units per layer increased for more complex models. • Used for all kinds of problem spaces. 18
- 19. Word Embedding Credit:Sebastian Ruder // sebastianruder.com • Projects sparse 1-hot vector representation onto denser lower dimensional space. • Unsupervised technique. • Embedding space exhibits Distributional Semantics. • Has almost completely replaced traditional distributional features in NLP (Deep Learning and non Deep Learning). • Word2Vec (CBOW and Skip-gram), GloVe. 19
- 20. Convolutional Neural Network • Alternate Convolution and Pooling Operations to extract relevant features • Mainly used in Image recognition, exploits geometry of image. • 1D variant (Convolution and Pooling) used for text. • Exploits word neighborhoods and extracts “meaning” of sentences or paragraphs. Credit: deeplearning.net 20
- 21. Recurrent Neural Network Credit: Andrej Karpathy // karpathy.github.io • Works with sequence input (such as text and audio). • Exploits temporal nature of the data. • Many variations possible (shown below). • Basic RNN suffers from vanishing gradient problem – addressed by Long Short Term Memory (LSTM) RNNs. • Gated Recurrent Unit (GRU) another variant with simpler structure and better performance than LSTM. 21
- 23. • Synthetic Dataset (1000, 10k, … records) • Composed of actors, places, things, actions, etc. • Released by Facebook Research bAbI Dataset • Single supporting fact • Two supporting facts • Three supporting facts • Two argument relations • Three argument relations • Yes/No questions • Counting • Lists/sets • Simple Negation • Indefinite Knowledge • Basic Coreference • Conjunction • Compound Coreference • Time Reasoning • Basic Deduction • Basic Induction • Positional Reasoning • Size Reasoning • Path Finding • Agent’s Motivations 23
- 24. bAbI Format (task 1) 24
- 25. bAbI LSTM • Implementation based on the paper: Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. • Test accuracy reported in paper: 50%. • Test accuracy achieved by implementation: 56%. • Code is here. 25
- 26. bAbI MemNN • Implementation based on the paper: End-to-end Memory Networks. • Test accuracy reported in paper: 99%. • Test accuracy achieved by implementation: 42%. • Code is here. 26
- 28. Data Format • 2000 multiple choice 8th Grade Science questions with 4 candidate answers and correct answer label. • 2000 questions without correct answer label. • Each question = 1 positive + 3 negative examples. • No story here. 28
- 29. QA-LSTM • Implementation based on the paper: LSTM- based Deep Learning Models for Non-factoid Answer Selection. • Test accuracy reported in paper: 64.3% (InsuranceQA dataset). • Test accuracy achieved by implementation: 56.93% (unidirectional) and 57% (bidirectional). • Code: unidirectional, bidirectional 29
- 30. Our Embedding Approach • 3 approaches to Embedding • Generate from Data • Use External model for lookup • Initialize with External model, then fine tune. • Not enough question data to generate good embedding. • Used pre-trained Google News word2vec model (trained with 3 billion words) • Model has 3 million word vectors of dimension (300,). • Uses gensim to read model. 30
- 31. QA-LSTM-CNN • Additional CNN Layer for more effective summarization. • Test accuracy reported in paper: 62.2% (InsuranceQA dataset). • Test accuracy achieved by implementation: 56.3% (unidirectional), did not try bidirectional. • Code is here. 31
- 32. Incorporating Attention • Vanishing Gradient problem addressed by LSTMs, but still shows up in long range Q+A contexts. • Solved using Attention Models • Based on visual models of human attention. • Allow the network to focus on certain words in question with “high resolution” and the rest at “low resolution”. • Similar to advice given for comprehension tests about reading the questions, then scanning passage for question keywords. • Implemented here as a dot product of question and answer, or question and story vectors. 32
- 33. QA-LSTM + Attention • Attention vector from question and answer combined with question. • Test accuracy reported in paper: 68.4% (InsuranceQA dataset). • Test accuracy achieved by implementation: 62.93% (unidirectional), 60.43% (bidirectional) • Code: unidirectional, bidirectional. 33
- 34. Incorporating External Knowledge • Contestants were allowed/advised to use external sources such as ConceptNet, CK-12 books, Quizlets, Flashcards from StudyStack, etc. • Significant crawling/scraping and parsing effort involved. • 4th place winner (tambietm) provides parsed download of StudyStack Flashcards on his Google drive. • Flashcard “story” = question || answer 34
- 35. Using Story Embedding • Build Word2Vec model using words from Flashcards. • Approximately 500k flashcards, 8,000 unique words. • Provides smaller, more focused embedding space. • Good performance boost over default Word2Vec embedding. 35 Model Default Embedding Story Embedding QA-LSTM with Attention 62.93 76.27 QA-LSTM Bidirectional with Attention 60.43 76.27
- 36. Relating Story to Question • Replicate bAbI setup: (story, question, answer). • Only a subset of flashcards relate to given question. • Using traditional IR methods to generate flashcard stories for each question. 36
- 37. QA-LSTM + Story • Story and Question combined to create Fact vector. • Question and Answer combined to create Attention vector • Fact and Attention vectors concatenated. • Test accuracy achieved by implementation: 70.47% (unidirectional), 61.77% (bidirectional). • Code: unidirectional, bidirectional. 37
- 38. Results Model Specifications Test Accuracy (%) QA-LSTM (Baseline) 56.93 QA-LSTM Bidirectional 57.0 QA-LSTM + CNN 55.7 QA-LSTM with Attention 62.93 QA-LSTM Bidirectional with Attention 60.43 QA-LSTM with Attention + Custom Embedding 76.27 * QA-LSTM Bidirectional w/Attention + Custom Embedding 76.27 * QA-LSTM + Attention + Story Facts 70.47 QA-LSTM Bidirectional + Attention + Story Facts 61.77 38
- 39. Model Deployment • Our models predict answer correct vs. incorrect. • Task is to choose the correct answer from candidate answers. • Re-instantiate trained model with Softmax layer removed. • Run batch of (story, question, answer) for each candidate answer. • Select best scoring answer as correct answer. 39
- 40. Deploying Model - Example 40
- 41. Future Work • Would like to implement Dynamic Memory Network (DMN) and Hierarchical Attention Based Convolutional Neural Network (HABCNN) models against this data. • Would like to submit to Kaggle to see how I did once Keras Issue 3927 is resolved. • Would like to try out these models against Stanford Question Answering Dataset (SQuAD) based on Wikipedia articles. • Would like to investigate Question Generation from text in order to generate training sets for Elsevier corpora. 41
- 42. Administrivia • Code for this talk:https://github.com/sujitpal/dl-models-for- qa • Contact me for questions and suggestions: sujit.pal@elsevier.com 42
- 43. Closing Thoughts • Deep Learning is rapidly becoming a general purpose solution for nearly all learning problems. • Information Retrieval approaches are still more successful on Question Answering than Deep Learning, but there are many efforts by Deep Learning researchers to change that. 43
- 44. Thank you. 44
Editor's Notes
- #11: Proposing the bAbI dataset.
- #12: Recurrent attention model over large external memory.
- #13: Closest to our use case, this is what our solution to the Kaggle comp is based on.
- #14: Socher’s dynamic memory models.
- #15: DMN, bAbI, Socher again
- #16: MCTest Benchmark, HABCNN
- #17: [1] bAbI dataset, [2] representation size (number of words), [3] MCTest Benchmark, Hierarchical Attention Based CNN (HABCNN), [4] DMN, Richard Socher [5]
- #23: [1] bAbI dataset, [2] representation size (number of words), [3] MCTest Benchmark, Hierarchical Attention Based CNN (HABCNN), [4] DMN, Richard Socher [5]
- #28: [1] bAbI dataset, [2] representation size (number of words), [3] MCTest Benchmark, Hierarchical Attention Based CNN (HABCNN), [4] DMN, Richard Socher [5]