BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2 likes1,009 views
The document presents BERT (Bidirectional Encoder Representations from Transformers), which utilizes deep bidirectional transformers to improve language understanding through unsupervised pre-training. It details the training methodologies, including masked language modeling and next sentence prediction, while highlighting its state-of-the-art performance across various NLP tasks. BERT serves as a baseline for future research, demonstrating the effectiveness of deep bidirectional architectures in natural language processing.
![My dog is hairy My dog is hairy
Choose 15% of tokens at random
80%
10%
10%
My dog is [Mask]
My dog is hairy
My dog is apple
Masked LM Procedure](https://image.slidesharecdn.com/bert-190114085327/85/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding-18-320.jpg)
![Task #2 - Next Sentence Prediction (NSP)
• Classification - [IsNext, NotNext]
• Final pre-trained model achieved 97-98%
accuracy.
!19](https://image.slidesharecdn.com/bert-190114085327/85/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding-19-320.jpg)
![!22
• The first token of every sequence is
always the special classification
embedding [CLS]. The final hidden state
corresponding to this token is used as the
aggregate sequence representation for
classification tasks. For non-classifcation
tasks, this vector is ignored.
• Sentence pairs are packed together into a
single sequence. The authors separate
them in two ways.
1. Separate with special token [SEP].
2. Add learned sentence embedding to
every token of corresponding
sentence.](https://image.slidesharecdn.com/bert-190114085327/85/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding-22-320.jpg)
![Differences between OpenAI GPT
!24
Model Corpus [CLS] / [SEP] tokens Steps Learning rate
BERT
BooksCorpus
+
Wikipedia
Learns during
pre-training
1M steps with batch
size of
128,000 words
Task-specific
fine-tuning
learning rate
OpenAI GPT BooksCorpus
Only introduced at
fine-tuning time
1M steps with batch
size of
32,000 words
Same learning rate of
5e-5](https://image.slidesharecdn.com/bert-190114085327/85/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding-24-320.jpg)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- 1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova !1 Google AI Language 2018.11.25 Presented by Young Seok Kimhttps://arxiv.org/abs/1810.04805
- 2. Articles & Useful Links • Official • ArXiv : https://arxiv.org/abs/1810.04805 • Blog : https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html • GitHub : https://github.com/google-research/bert • Unofficial • Lyrn.ai blog : https://www.lyrn.ai/2018/11/07/explained-bert-state-of-the-art- language-model-for-nlp/ • Korean blog : https://rosinality.github.io/2018/10/bert-pre-training-of-deep- bidirectional-transformers-for-language-understanding !2
- 3. Related Papers • Vaswani, Ashish et al. “Attention Is All You Need.” NIPS (2017) • PR-049 : https://youtu.be/6zGgVIlStXs • Tutorial with code : http://nlp.seas.harvard.edu/2018/04/03/attention.html • Radford, Alec. “Improving Language Understanding by Generative Pre-Training.” (2018) • Website : https://blog.openai.com/language-unsupervised/ • Paper : https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language- unsupervised/language_understanding_paper.pdf • Wang, Alex et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” (2018) • Website : https://gluebenchmark.com/ • Paper : https://arxiv.org/abs/1804.07461 !3
- 5. Attention is All you need • Introduced Transformer module • Reduced computational complexity in respect to the sequence length !5
- 6. GLUE • Benchmark introduced in Wang, Alex et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” (2018) • Contains 11 Tasks !6
- 7. BERT Bidirectional Encoder Representations from Transformers !7
- 8. Motivation !8 Traditional RNN / LSTM / GRU units
- 10. Motivation
- 11. Problem • Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “see itself” in a multi-layered context. !11
- 12. Problem E 1 T 1 E 2 … EN Transformer Transformer Transformer… T 2 TN … Single Transformer Layer
- 13. !13 E 1 T 1 E 2 … E N Transformer Transformer Transformer… T2 TN Transformer Transformer Transformer… … Multi-layer Transformer Layer
- 14. !14 E 1 T 1 E 2 … E N Transformer Transformer Transformer… T2 TN Transformer Transformer Transformer… … Multi-layer Transformer Layer
- 15. Training Method Task #1 - Masked Language Model (MLM) Task #2 - Next Sentence Prediction (NSP)
- 16. Task #1 - Masked LM !16 • Fill in the blank! • Formally, Cloze Test (https://en.wikipedia.org/wiki/Cloze_test) • Similar to CBOW in Word2Vec?
- 17. Task #1 - Masked LM !17 • Fill in the blank! • Formally, Cloze Test (https://en.wikipedia.org/wiki/Cloze_test) • Similar to CBOW in Word2Vec?
- 18. My dog is hairy My dog is hairy Choose 15% of tokens at random 80% 10% 10% My dog is [Mask] My dog is hairy My dog is apple Masked LM Procedure
- 19. Task #2 - Next Sentence Prediction (NSP) • Classification - [IsNext, NotNext] • Final pre-trained model achieved 97-98% accuracy. !19
- 20. Embedding !20
- 21. !21
- 22. !22 • The first token of every sequence is always the special classification embedding [CLS]. The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks. For non-classifcation tasks, this vector is ignored. • Sentence pairs are packed together into a single sequence. The authors separate them in two ways. 1. Separate with special token [SEP]. 2. Add learned sentence embedding to every token of corresponding sentence.
- 23. Corpus • BookCorpus (800M words) • English Wikipedia (2500M words) • Training dataset • 50% - Two adjacent sentences • 50% - Random sentence after a sentence. !23
- 24. Differences between OpenAI GPT !24 Model Corpus [CLS] / [SEP] tokens Steps Learning rate BERT BooksCorpus + Wikipedia Learns during pre-training 1M steps with batch size of 128,000 words Task-specific fine-tuning learning rate OpenAI GPT BooksCorpus Only introduced at fine-tuning time 1M steps with batch size of 32,000 words Same learning rate of 5e-5
- 25. Results !25
- 27. GLUE Benchmark • MNLI: Multi-Genre Natural Language Inference • Given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction, or neutral with respect to the first sentence. • Two versions - MNLI matched, MNLI mismatched • Two sentence, classification task !27
- 28. GLUE Benchmark • QQP: Quora Question Pairs • Quora Question Pairs is a binary classification task where the goal is to determine if two questions asked on Quora are semantically equivalent • Two sentence, binary classification task !28
- 29. GLUE Benchmark • QNLI: Question Natural Language Inference • The positive examples are (question, sentence) pairs which do contain the correct answer, and the negative examples are (question, sentence) from the same paragraph which do not contain the answer. • Two sentence, binary classification task !29
- 30. GLUE Benchmark • SST-2: Stanford Sentiment Treebank • Binary single-sentence classification task consisting of sentences extracted from movie reviews with human annotations of their sentiment • One sentence, binary classification task !30
- 31. GLUE Benchmark • CoLA: Corpus of Linguistic Acceptability • Binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not • One sentence, binary classification task !31
- 32. GLUE Benchmark • STS-B: The Semantic Textual Similarity Bench- mark • Binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not • One sentence, binary classification task !32
- 33. GLUE Benchmark • MRPC: Microsoft Research Paraphrase Corpus • Consists of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent • Two sentence, binary classification task !33
- 34. GLUE Benchmark • RTE: Recognizing Textual Entailment • A binary entailment task similar to MNLI, but with much less training data • Two sentence, binary classification task !34
- 35. GLUE Benchmark • WNLI: Winograd Natural Language Inference • A binary entailment task similar to MNLI, but with much less training data • The GLUE webpage notes that there are issues with the construction of this dataset • Authors therefore exclude this set !35
- 36. !36
- 38. SQuAD v1.1 • Standford Question Answering Dataset (SQuAD) is a collection of 100k crowdsourced question/answer pairs • Given a question and a paragraph from Wikipedia containing the answer, the task is to predict the answer text span in the paragraph !38
- 39. Results on SQuAD v1.1 !39
- 40. SWAG • Situations With Adversarial Generations Dataset • Given a sentence from a video captioning dataset, the task is to decide among four choices the most plausible continuation. !40
- 41. SWAG Results
- 42. GLUE Results
- 44. Model size !44
- 45. Conclusion • Unsupervised pre-training is now an integral part of many language understanding systems. • Now models can be truly trained with deep bidirectional architectures. • State-of-the-art on almost every NLP tasks, in some cases surpassing human performance. !45
- 46. Personal thoughts • Paper is well written and easy to follow • SOTA in not just one task/dataset but in almost all tasks • I think this method is going to be used universally as a baseline for future NLP research • More objective comparison between BERT and OpenAI GPT was possible because the baseline parameters are chosen such that it is almost identical to OpenAI GPT • Model looks very simple but at the same time very flexible to adapt towards various tasks with simple modifications on the top layer • Unsupervised pre-training and supervised fine-tuning might prevail in many domain. !46
- 47. Thank you! !47
- 48. References • Images are either from • original papers or • https://towardsdatascience.com/introduction-to-sequence-models-rnn-bidirectional- rnn-lstm-gru-73927ec9df15 • https://colah.github.io/posts/2015-08-Understanding-LSTMs/ • https://www.lyrn.ai/2018/11/07/explained-bert-state-of-the-art-language-model-for- nlp/ !48