Transformers and BERT with SageMaker
1 like356 views
The document provides an overview of Transformers and BERT models for natural language processing tasks. It explains that Transformers use self-attention mechanisms to overcome limitations of RNNs in capturing long-term dependencies. The encoder-decoder architecture is described, with the encoder generating representations and the decoder generating target sequences. Key aspects like multi-head attention, positional encoding, and pre-training are summarized. The document details how BERT is pretrained using masked language modeling and next sentence prediction to learn contextual representations. It shows how BERT can then be fine-tuned for downstream tasks like sentiment analysis and named entity recognition.
![How BERT works ?
[CLS] Kolkata is a beautiful [MASK] [SEP] I love Kolkata [SEP]
Encoder Layer 1
Encoder Layer 1
Encoder Layer 12
E[CLS] EKolkata Eis Ea Ebeautiful E[SEP] EI Elove EKolkata E[SEP]
Token Embeddings
EA EA EA EA EA EA EB EB EB EB
Segment Embeddings
E0 E1 E2 E3 E4 E6 EB7 E8 E9 E10
Position Embeddings
INPUT
E[MASK]
EA
E5
R[CLS] RKolkata Ris Ra Rbeautiful R[MASK] R[SEP] R[SEP]
RI Rlove RKolkata
OUTPUT
(Enhanced Embedding)
Pre-Training Tasks](https://image.slidesharecdn.com/transformers-210429155830/85/Transformers-and-BERT-with-SageMaker-31-320.jpg)
![How BERT works ?
Pre-Training & Fine-Tuning
[CLS] Suman loves Kolkata [SEP]
Encoder Layer 1
Encoder Layer 1
Encoder Layer 12
INPUT
R[CLS] R[SEP]
RSuman Rloves RKolkata
OUTPUT
(Enhanced Embedding)](https://image.slidesharecdn.com/transformers-210429155830/85/Transformers-and-BERT-with-SageMaker-32-320.jpg)
![How BERT works ?
Pre-Training & Fine-Tuning
.9
.1
Positive
Negative
FFN
+
Softmax
(Sentiment Analysis)
[CLS] Suman loves Kolkata [SEP]
Encoder Layer 1
Encoder Layer 1
Encoder Layer 12
INPUT
R[CLS] R[SEP]
RSuman Rloves RKolkata
OUTPUT
(Enhanced Embedding)](https://image.slidesharecdn.com/transformers-210429155830/85/Transformers-and-BERT-with-SageMaker-33-320.jpg)
![How BERT works ?
Pre-Training & Fine-Tuning
[CLS] Suman loves Kolkata [SEP]
Encoder Layer 1
Encoder Layer 1
Encoder Layer 12
INPUT
OUTPUT
(Enhanced Embedding)
R[CLS] R[SEP]
RSuman Rloves RKolkata
Classifier
PERSON LOCATION
(Name entity recognition)
.9
.1
Positive
Negative
FFN
+
Softmax
(Sentiment Analysis)
[CLS] Suman loves Kolkata [SEP]
Encoder Layer 1
Encoder Layer 1
Encoder Layer 12
INPUT
R[CLS] R[SEP]
RSuman Rloves RKolkata
OUTPUT
(Enhanced Embedding)](https://image.slidesharecdn.com/transformers-210429155830/85/Transformers-and-BERT-with-SageMaker-34-320.jpg)
Transformers and BERT with SageMaker
- 1. Inside Transformers and BERT Suman Debnath Principal Developer Advocate, India
- 2. Why Transformers? • RNN and LSTM • Machine Translation • Text Generation • Next word prediction • More… • Sequential processing and learning • Challenge • Recurrent Model • Long-term dependency • Bidirectional context is not taken into account • Context understanding « The bat flew past my window » vs. « He hit the baseball with the bat »
- 3. • Overcome this limitation of RNNs • SOTA model for several NLP tasks • Paved the way for new revolutionary architectures such as BERT, GPT-3 and more • Based entirely on the attention mechanism and completely gets rid of recurrence Attention Is All You Need
- 4. Let's understand how the transformer works (language translation task) • Encoder-Decoder architecture • Feed the input sentence (source sentence) to the encoder • Encoder learns the representation of the input sentence and sends the representation to the decoder • The decoder receives the representation learned by the encoder as input and generates the output sentence (target sentence). Encoder Decoder I am good je vais bien Representation
- 5. The encoder of the transformer • Stack of N number of encoders • The output of one encoder is sent as input to the encoder above it. • Questions? • How exactly does the encoder work ? • How is it generating the representation for the given source sentence (input sentence)? Encoder Layer 1 Encoder Layer 2 I am good Representation Encoder Layer N
- 6. How exactly does the encoder work ? • All the encoder blocks are identical • Each encoder block consists of two sublayers • Multi-head attention • Feedforward network(FNN) Encoder Layer 2 Encoder Layer 1 FFN FFN Representation Multi-head attention Multi-head attention I am good
- 7. Self-attention mechanism • The pronoun it could mean either dog or food • We know, it implies the dog and not food • But how our model can understand ? • Here is where the self-attention mechanism helps us • Model relates the word it to ALL the words in the sentence “A dog ate the food because it was hungry” How exactly does this work?
- 8. I am good • The embedding of the word I is : • The embedding of the word am is : • The embedding of the word good is : *embedding dimension be 512
- 9. 3 New Matrices : {query, key, value} Q K V WQ WK WV Weight Matrices WQ WK WV (randomly initialized, learned during training)
- 10. 3 New Matrices : {query, key, value} Q K V WQ WK WV Weight Matrices WQ WK WV (randomly initialized, learned during training) Implies the query, key, and value vectors of the word “I” *the dimension of the {query, key, value} vector(dk) is 64.
- 11. Why are we computing this? What is the use of query, key, and value matrices? How is this going to help us?
- 12. Self-attention mechanism • We learnt to compute matrices : query(Q), key(K), and value(V) • They are obtained from the Input matrix, X • How they are used in the self-attention mechanism • REMEMBER: self-attention mechanism relates the word to all the words in the given sentence • This understanding helps to learn better representation • Next, we will see how these 3 matrices, helps to get a better representation
- 13. 4 Steps
- 14. Step 1 (”Dot product” between the query matrix, Q, and the key matrix, KT ) What exactly Q.KT does signify? “how similar they are”
- 15. Step 2 (”Divide” the matrix by the square root of the dimension of the key vector) Why this is needed? “useful in obtaining stable gradients” *the dimension of the key vector(dk) is 64.
- 16. Step 3 (”Normalize” the matrix by the square root of the dimension of the key vector) We normalize them using the softmax function (bringing the score in the range of 0 to 1 and the sum equals to 1) Word “I” is related to: - itself by 90% - am by 7% - good by 3%
- 17. Step 4 (final step in the self-attention mechanism is to compute the “attention matrix”) The attention matrix(Z) contains the attention values for each word in the sentence (sum of the value vectors weighted by the scores)
- 18. Self-attention of the word “I” is computed as the sum of the value vectors weighted by the scores
- 19. Using a self attention mechanism, we can understand how a word is related to all other words in the sentence. Step 1 Step 2 Step 3 Step 4
- 20. Let’s recall Encoder Layer 2 Encoder Layer 1 FFN FFN I am good Representation Multi-head attention Multi-head attention
- 21. Multi-head attention mechanism • Instead of having a single attention head, we can use multiple attention heads • Instead of computing a single attention matrix, Z, we can compute multiple attention matrices • This will be useful only in circumstances where the meaning of the actual word is ambiguous, e.g. “A dog ate the food because it was hungry”
- 22. Positional encoding • Transformer network, we don't follow the recurrence mechanism • How will it understand the meaning of the sentence if the word order is not retained ? • We should give some information about the word order to the transformer so that it can understand the sentence + =
- 23. Positional encoding -1 -0.25 0.91 -0.25 Positional Encoding for the 30th Word Dim 1 Dim 2 Dim 3 Dim 4
- 24. Let’s recall FFN FFN I am good Representation Multi-head attention Multi-head attention FFN I am good Representation Multi-head attention Positional Encoding
- 25. Feedforward network • The feedforward network consists of two dense layers with ReLU activations • The parameters of the feedforward network are the same over the different positions of the sentence and different over the encoder blocks.
- 26. Add and norm component • It connects the input and output of a sublayer. • Connects the input of the multi-head attention sublayer to its output • Connects the input of the feedforward sublayer to its output • Residual connection followed by Layer normalization • Layer normalization promotes faster training by preventing the values in each layer from changing heavily
- 28. Encoder Decoder I am good je vais bien Representation
- 29. BERT(Bidirectional Encoder Representation from Transformer) Multi-head attention Multi-head attention Multi-head attention FFN Encoder 1 Python is my favorite programming language Encoder N Encoder 2 RPython Ris Rmy Rfavorite Rprogramming Rlanguage
- 30. • BERT is supposed to do: • Masked Language Model (MLM) • Next Sentence Prediction (NSP) • BERT was trained to perform these two tasks purely as a way to force it to develop a sophisticated understanding of language, e.g. “Kolkata is a beautiful city. I love Kolkata” • Here’s what BERT is supposed to do: • MLM - Predict the crossed out word (Correct answer is “city”). • NSP - Was sentence B found immediately after sentence A , or from somewhere else? (Correct answer is that they are consecutive). How BERT works ? Pre-Training Tasks
- 31. How BERT works ? [CLS] Kolkata is a beautiful [MASK] [SEP] I love Kolkata [SEP] Encoder Layer 1 Encoder Layer 1 Encoder Layer 12 E[CLS] EKolkata Eis Ea Ebeautiful E[SEP] EI Elove EKolkata E[SEP] Token Embeddings EA EA EA EA EA EA EB EB EB EB Segment Embeddings E0 E1 E2 E3 E4 E6 EB7 E8 E9 E10 Position Embeddings INPUT E[MASK] EA E5 R[CLS] RKolkata Ris Ra Rbeautiful R[MASK] R[SEP] R[SEP] RI Rlove RKolkata OUTPUT (Enhanced Embedding) Pre-Training Tasks
- 32. How BERT works ? Pre-Training & Fine-Tuning [CLS] Suman loves Kolkata [SEP] Encoder Layer 1 Encoder Layer 1 Encoder Layer 12 INPUT R[CLS] R[SEP] RSuman Rloves RKolkata OUTPUT (Enhanced Embedding)
- 33. How BERT works ? Pre-Training & Fine-Tuning .9 .1 Positive Negative FFN + Softmax (Sentiment Analysis) [CLS] Suman loves Kolkata [SEP] Encoder Layer 1 Encoder Layer 1 Encoder Layer 12 INPUT R[CLS] R[SEP] RSuman Rloves RKolkata OUTPUT (Enhanced Embedding)
- 34. How BERT works ? Pre-Training & Fine-Tuning [CLS] Suman loves Kolkata [SEP] Encoder Layer 1 Encoder Layer 1 Encoder Layer 12 INPUT OUTPUT (Enhanced Embedding) R[CLS] R[SEP] RSuman Rloves RKolkata Classifier PERSON LOCATION (Name entity recognition) .9 .1 Positive Negative FFN + Softmax (Sentiment Analysis) [CLS] Suman loves Kolkata [SEP] Encoder Layer 1 Encoder Layer 1 Encoder Layer 12 INPUT R[CLS] R[SEP] RSuman Rloves RKolkata OUTPUT (Enhanced Embedding)
- 35. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo
- 36. Few great books • Resources • Data Science on AWS • Getting Started with Google BERT
- 37. Suman Debnath Developer Advocate, India /in/suman-d/
- 38. The decoder of the transformer • Stake of N number of decoders • The output of one decoder is sent as input to the decoder above it • Decoder receives two inputs: • The previous decoder’s output • The encoder's representation • Question? • How exactly does the decoder generate the target sentence?
- 39. At time step t=1 At time step t=2 At time step t=3 Similarly, on every time step, the decoder combines the newly generated word to the input and predicts the next word. At time step t=4
- 40. How exactly does the decoder work? • All the encoder blocks are identical • Each encoder block consists of three sublayers • Masked multi-head attention • Multi-head attention • Feedforward network(FNN) Decoder Block FFN Representation Masked multi-head attention Multi-head attention
- 41. Masked multi-head attention • During training: • Since we have the right target sentence • We can just feed the whole target sentence as input to the decoder but with a small modification • During testing: • Decoder predicts the target sentence word by word in each time step • We learned that the decoder takes the input <sos> as the first token • Keep predicting the target sentence until the <eos> token is reached Training Data set
- 44. Attention Matrix Encoder Representation
- 46. Feedforward network • The feedforward layer in the decoder works exactly the same as what we learned in the encoder
- 47. Add and norm component • The add and norm component in the decoder works exactly the same as what we learned in the encoder
- 48. Linear and softmax layers • Feed the output obtained from the topmost decoder to the linear and softmax layers
- 50. The complete transformer architecture