[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD
4 likes668 views
Adams Wei Yu is a PhD candidate at CMU working on machine reading comprehension and large scale optimization. His advisors are Jaime Carbonell and Alex Smola. He has worked on question answering models and datasets like SQuAD. QANet is one of his contributions, which uses self-attention and convolutional layers instead of RNNs for question answering. It achieves state-of-the-art results while being much faster to train and run than previous models.
![[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-12-320.jpg)
![[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-30-320.jpg)
![General framework neural QA Systems
Bi-directional Attention Flow (BiDAF)
[Seo et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-33-320.jpg)
![Base Model (BiDAF)
Similar general architectures:
● R-Net [Wang et al, ACL’17]
● DCN [Xiong et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-34-320.jpg)
![Base Model (BiDAF)
Similar general architectures:
● R-Net [Wang et al, ACL’17]
● DCN [Xiong et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-35-320.jpg)
![Base Model (BiDAF)
36
Similar general architectures:
● R-Net [Wang et al, ACL’17]
● DCN [Xiong et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-36-320.jpg)
![Base Model (BiDAF)
Similar general architectures:
● R-Net [Wang et al, ACL’17]
● DCN [Xiong et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-37-320.jpg)
![Base Model (BiDAF)
Similar general architectures:
● R-Net [Wang et al, ACL’17]
● DCN [Xiong et al., ICLR’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-38-320.jpg)
![The todayniceisweather
The todayniceisweather
0.6
0.2
0.8
0.4
0.1
0.6
0.4
0.1
0.4
0.9
0.1
0.8
0.2
0.3
0.1
The todayniceisweather
w1
x + w2
x + w3
x + w4
x + w5
x
1.8
2.3
0.4
The
=
w1 w2
w3
w4
w5
w1
, w2
, w3
, w4
, w5
= softmax ( )
0.6
0.2
0.8
0.4
0.1
0.6
0.4
0.1
0.4
0.9
0.1
0.8
0.2
0.3
0.1
0.6 0.2 0.8 x
The
The todayniceisweather
[Vaswani et al., NIPS’17]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-50-320.jpg)
![Position Emb
Feedforward
Layer Norm
Self Attention
Layer Norm
Convolution
Layer Norm
+
+
+
Repeat
Position Emb
Feedforward
Self Attention
Repeat
Convolution
if you want to
go deeper
QANet Encoder
[Yu et al., ICLR’18]](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-54-320.jpg)
![[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-66-320.jpg)
![Transfer learning for richer presentation
● Pretrained language model
(ELMo, [Peters et al., NAACL’18])
○ + 4.0 F1](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-70-320.jpg)
![Transfer learning for richer presentation
71
● Pretrained language model
(ELMo, [Peters et al., NAACL’18])
○ + 4.0 F1
● Pretrained machine translation
model (CoVe [McCann, NIPS’17])
○ + 0.3 F1](https://image.slidesharecdn.com/246qanetdeview2018-181012000849/85/246-QANet-Towards-Efficient-and-Human-Level-Reading-Comprehension-on-SQuAD-71-320.jpg)
[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD
- 1. Adams Wei Yu Deview 2018, Seoul Quoc Le Thang Luong Rui Zhao Mohammad Norouzi Kai Chen Collaborators David Dohan
- 2. Bio Adams Wei Yu ● Ph.D Candidate @ MLD, CMU ○ Advisor: Jaime Carbonell, Alex Smola ○ Large scale optimization ○ Machine reading comprehension
- 3. Question Answering Concrete Answer No clear answer
- 5. http://www.aaai.org/Magazine/Watson/watson.php Watson: complex multi-stage system
- 6. Moving towards end-to-end systems ● Translation ● Question Answering
- 7. Lots of Datasets Available TriviaQA Narrative QA MS Marco
- 8. Stanford Question Answer Dataset (SQuAD) In education, teachers facilitate student learning, often in a school or academy or perhaps in another environment such as outdoors. A teacher who teaches on an individual basis may be described as a tutor. Passage: What is the role of teachers in education?Question: facilitate student learningGroundtruth: facilitate student learningPrediction 1: EM = 1, F1 = 1 student learningPrediction 2: EM = 0, F1 = 0.8 teachers facilitate student learningPrediction 3: EM = 0, F1 = 0.86 Data: Crowdsourced 100k question-answer pairs on 500 Wikipedia articles.
- 9. Roadmap ● Models for text ● General neural structures for QA ● Building blocks for QANet ○ Fully parallel (CNN + Self-attention) ○ data augmentation via back-translation ○ transfer learning from unsupervised tasks
- 10. That movie was awful.
- 11. That movie was awful . embed embed embed embed embed sum Bag of words hout
- 13. Continuous bag-of-words and skip-gram architectures (Mikolov et al., 2013a; 2013b)
- 14. That movie was awful . conv conv conv conv conv sum embed embed embed embed embed Bag of N-Grams hout
- 15. That movie was awful . embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h) hinit hout Recurrent Neural Networks
- 16. The quick brown fox jumped over the lazy doo
- 17. The quick brown fox jumped over the lazy dog
- 18. A feed-forward neural network language model (Bengio et al., 2001; 2003)
- 19. <s> The quick brown fox embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h)hinit project The quick brown fox jumped Language Models
- 20. Yes please <de> ja bitte embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h)hinit project Ja bitte </s> Language Models Seq2Seq
- 21. Yes please <s> ja bitte embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h)hinit Ja bitte </s> Seq2Seq + Attention Encoder Decoder hinit ?
- 24. Attention: a weighted average The cat stuck out its tongue and licked its owner The cat stuck out its tongue and licked its owner
- 25. Convolution: Different linear transformations by relative position. The cat stuck out its tongue and licked its owner The cat stuck out its tongue and licked its owner
- 26. Attention: a weighted average The cat stuck out its tongue and licked its owner The cat stuck out its tongue and licked its owner
- 27. Multi-head Attention Parallel attention layers with different linear transformations on input and output. The cat stuck out its tongue and licked its owner The cat stuck out its tongue and licked its owner
- 28. Yes please <s> Ja embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h)hinit Ja ...bitte </s> Seq2Seq + Attention Encoder Decoder hinit w1 w2
- 29. <s> The quick brown fox embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h) project The quick brown fox jumped Language Models with attention
- 31. Roadmap ● Models for text ● General neural structures for QA ● Building blocks for QANet ○ Fully parallel (CNN + Self-attention) ○ data augmentation via back-translation ○ transfer learning from unsupervised tasks
- 32. General (Doc, Question) → Answer Model
- 33. General framework neural QA Systems Bi-directional Attention Flow (BiDAF) [Seo et al., ICLR’17]
- 34. Base Model (BiDAF) Similar general architectures: ● R-Net [Wang et al, ACL’17] ● DCN [Xiong et al., ICLR’17]
- 35. Base Model (BiDAF) Similar general architectures: ● R-Net [Wang et al, ACL’17] ● DCN [Xiong et al., ICLR’17]
- 36. Base Model (BiDAF) 36 Similar general architectures: ● R-Net [Wang et al, ACL’17] ● DCN [Xiong et al., ICLR’17]
- 37. Base Model (BiDAF) Similar general architectures: ● R-Net [Wang et al, ACL’17] ● DCN [Xiong et al., ICLR’17]
- 38. Base Model (BiDAF) Similar general architectures: ● R-Net [Wang et al, ACL’17] ● DCN [Xiong et al., ICLR’17]
- 39. RNN RNN RNN RNN Two Challenges with RNNs Remain... Base Model (BiDAF)
- 40. First challenge: hard to capture long dependency h1 h3 h4 h5 h6 h2 Being a long-time fan of Japanese film, I expected more than this. I can't really be bothered to write too much, as this movie is just so poor. The story might be the cutest romantic little something ever, pity I couldn't stand the awful acting, the mess they called pacing, and the standard "quirky" Japanese story. If you've noticed how many Japanese movies use characters, plots and twists that seem too "different", forcedly so, then steer clear of this movie. Seriously, a 12-year old could have told you how this movie was going to move along, and that's not a good thing in my book. Fans of "Beat" Takeshi: his part in this movie is not really more than a cameo, and unless you're a rabid fan, you don't need to suffer through this waste of film.
- 41. Second challenge: hard to compute in parallel Strictly Sequential!
- 42. 1. local context input hidden state h1 h3 h4 h5 h6 h2 2. global interaction 3. Temporal info What do RNNs Capture? Substitution?
- 43. Roadmap ● Models for text ● General neural structures for QA ● Building blocks for QANet ○ Fully parallel (CNN + Self-attention) ○ data augmentation via back-translation ○ transfer learning from unsupervised tasks
- 44. Convolution: Capturing Local Context 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 The todayniceisweather
- 45. Convolution: Capturing Local Context 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.4 0.72.51.10.6 k = 2 d = 3 0.0 0.0 0.0
- 46. Convolution: Capturing Local Context 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.4 0.72.51.10.6 k = 2 d = 3 0.0 0.0 0.0
- 47. Convolution: Capturing Local Context 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.4 0.72.51.10.6 k = 2 d = 3 0.0 0.0 0.0
- 48. Convolution: Capturing Local Context 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.4 0.72.51.10.6 1.8 0.90.30.41.6 k = 3k = 2 d = 3 1.2 0.81.40.52.1 k = 3 0.0 0.0 0.0 k-gram features Fully parallel!
- 49. How about Global Interaction? The todayniceisweather layer 1 layer 2 layer 3 1. May need O(logk N) layers 2. Interaction may become weaker N: Seq length. k: Filter size.
- 50. The todayniceisweather The todayniceisweather 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 The todayniceisweather w1 x + w2 x + w3 x + w4 x + w5 x 1.8 2.3 0.4 The = w1 w2 w3 w4 w5 w1 , w2 , w3 , w4 , w5 = softmax ( ) 0.6 0.2 0.8 0.4 0.1 0.6 0.4 0.1 0.4 0.9 0.1 0.8 0.2 0.3 0.1 0.6 0.2 0.8 x The The todayniceisweather [Vaswani et al., NIPS’17]
- 51. The todayniceisweather The todayniceisweather Self-attention is fully parallel & all-to-all!
- 52. Per Unit Total Per Layer Sequential Op (Path Memory) Self-Attn O(Nd) O(N2 d) O(1) Conv O(kd2 ) O(kNd2 ) O(1) RNN O(d2 ) O(Nd2 ) O(N) Complexity Self-Attn Conv RNN N: Seq length. d: Dim. (N > d) k: Filter size.
- 53. Explicitly Encode Temporal Info + ++++ RNN Position Embedding 1 5432 1 5432 Implicit encode explicit encode
- 54. Position Emb Feedforward Layer Norm Self Attention Layer Norm Convolution Layer Norm + + + Repeat Position Emb Feedforward Self Attention Repeat Convolution if you want to go deeper QANet Encoder [Yu et al., ICLR’18]
- 55. RNN RNN RNN RNN Base Model (BiDAF) → QANet QANet Encoder QANet Encoder QANet Encoder QANet Encoder
- 56. 130 layers QANet – 130+ layers (Deepest NLP NN)
- 57. QANet – First QA system with No Recurrence ● Very fast! ○ Training: 3x - 13x ○ Inference: 4x - 9x
- 58. QANet – 130+ layers (Deepest NLP NN) ● Layer normalization ● Residual connections ● L2 regularization ● Stochastic Depth ● Squeeze and Excitation ● ...
- 59. Roadmap ● Models for text ● General neural structures for QA ● Building blocks for QANet ○ Fully parallel (CNN + Self-attention) ○ data augmentation via back-translation ○ transfer learning from unsupervised tasks
- 60. Data augmentation: popular in vision & speech
- 61. More data with NMT back-translation Input Paraphrase Translation English → French English ← French Previously, tea had been used primarily for Buddhist monks to stay awake during meditation. Autrefois, le thé avait été utilisé surtout pour les moines bouddhistes pour rester éveillé pendant la méditation. In the past, tea was used mostly for Buddhist monks to stay awake during the meditation.
- 62. More data with NMT back-translation Input Paraphrase Translation English → French English ← French Previously, tea had been used primarily for Buddhist monks to stay awake during meditation. In the past, tea was used mostly for Buddhist monks to stay awake during the meditation. ● More data ○ (Input, label) ○ (Paraphrase, label) Applicable to virtually any NLP tasks!
- 63. QANet augmentation Input Paraphrase Translation English → French English ← French Improvement: +1.1 F1 Use 2 language pairs: English-French, English-German. 3x data.
- 64. Roadmap ● Models for text ● General neural structures for QA ● Building blocks for QANet ○ Fully parallel (CNN + Self-attention) ○ data augmentation via back-translation ○ transfer learning from unsupervised tasks
- 67. Transfer learning for richer presentation
- 68. <s> The quick brown fox embed embed embed embed embed f(x,h) f(x,h) f(x,h) f(x,h) f(x,h)hinit project The quick brown fox jumped Language Models
- 69. Sebastian Ruder @ Indaba 2018
- 70. Transfer learning for richer presentation ● Pretrained language model (ELMo, [Peters et al., NAACL’18]) ○ + 4.0 F1
- 71. Transfer learning for richer presentation 71 ● Pretrained language model (ELMo, [Peters et al., NAACL’18]) ○ + 4.0 F1 ● Pretrained machine translation model (CoVe [McCann, NIPS’17]) ○ + 0.3 F1
- 72. QANet – 3 key ideas ● Deep Architecture without RNN ○ 130-layer (Deepest in NLP) ● Transfer Learning ○ leverage unlabeled data ● Data Augmentation ○ with back-translation #1 on SQuAD (Mar-Aug 2018)
- 73. QA is not Solved!!
- 74. QA is not Solved!! Thank you!