SlideShare a Scribd company logo
Natural Language Processing
with Deep Learning
CS224N/Ling284
John Hewitt
Lecture 10: Pretraining
Lecture Plan
1. A brief note on subword modeling
2. Motivating model pretraining from word embeddings
3. Model pretraining three ways
1. Decoders
2. Encoders
3. Encoder-Decoders
4. Interlude: what do we think pretraining is teaching?
5. Very large models and in-context learning
Reminders:
Assignment 5 is out today! It covers lecture 9 (Tuesday) and lecture 10 (Today)!
It has ~pedagogically relevant math~ so get started!
2
Word structure and subword models
Let’s take a look at the assumptions we’ve made about a language’s vocabulary.
We assume a fixed vocab of tens of thousands of words, built from the training set.
All novel words seen at test time are mapped to a single UNK.
word vocab mapping embedding
hat → pizza (index)
learn → tasty (index)
taaaaasty → UNK (index)
laern → UNK (index)
Transformerify → UNK (index)
3
Common
words
Variations
misspellings
novel items
Word structure and subword models
Finite vocabulary assumptions make even less sense in many languages.
• Many languages exhibit complex morphology, or word structure.
• The effect is more word types, each occurring fewer times.
4
Example: Swahili verbs can have
hundreds of conjugations, each
encoding a wide variety of
information. (Tense, mood,
definiteness, negation, information
about the object, ++)
Here’s a small fraction of the
conjugations for ambia – to tell.
[Wiktionary]
The byte-pair encoding algorithm
Subword modeling in NLP encompasses a wide range of methods for reasoning about
structure below the word level. (Parts of words, characters, bytes.)
• The dominant modern paradigm is to learn a vocabulary of parts of words (subword tokens).
• At training and testing time, each word is split into a sequence of known subwords.
Byte-pair encoding is a simple, effective strategy for defining a subword vocabulary.
1. Start with a vocabulary containing only characters and an “end-of-word” symbol.
2. Using a corpus of text, find the most common adjacent characters “a,b”; add “ab” as a subword.
3. Replace instances of the character pair with the new subword; repeat until desired vocab size.
Originally used in NLP for machine translation; now a similar method (WordPiece) is used in pretrained
models.
5 [Sennrich et al., 2016, Wu et al., 2016]
Word structure and subword models
Common words end up being a part of the subword vocabulary, while rarer words are split
into (sometimes intuitive, sometimes not) components.
In the worst case, words are split into as many subwords as they have characters.
word vocab mapping embedding
hat → hat
learn → learn
taaaaasty → taa## aaa## sty
laern → la## ern##
Transformerify → Transformer## ify
6
Common
words
Variations
misspellings
novel items
Outline
1. A brief note on subword modeling
2. Motivating model pretraining from word embeddings
3. Model pretraining three ways
1. Decoders
2. Encoders
3. Encoder-Decoders
4. Interlude: what do we think pretraining is teaching?
5. Very large models and in-context learning
7
Motivating word meaning and context
Recall the adage we mentioned at the beginning of the course:
“You shall know a word by the company it keeps” (J. R. Firth 1957: 11)
This quote is a summary of distributional semantics, and motivated word2vec. But:
“… the complete meaning of a word is always contextual,
and no study of meaning apart from a complete context
can be taken seriously.” (J. R. Firth 1935)
Consider I record the record: the two instances of record mean different things.
8 [Thanks to Yoav Goldberg on Twitter for pointing out the 1935 Firth quote.]
Where we were: pretrained word embeddings
Circa 2017:
• Start with pretrained word embeddings (no
context!)
• Learn how to incorporate context in an LSTM
or Transformer while training on the task.
Some issues to think about:
• The training data we have for our
downstream task (like question answering)
must be sufficient to teach all contextual
aspects of language.
• Most of the parameters in our network are
randomly initialized!
9
… the movie was …
ෝ
𝒚
Not pretrained
pretrained
(word embeddings)
[Recall, movie gets the same word embedding,
no matter what sentence it shows up in]
Where we’re going: pretraining whole models
In modern NLP:
• All (or almost all) parameters in NLP
networks are initialized via pretraining.
• Pretraining methods hide parts of the input
from the model, and train the model to
reconstruct those parts.
• This has been exceptionally effective at
building strong:
• representations of language
• parameter initializations for strong NLP
models.
• Probability distributions over language that
we can sample from
10
… the movie was …
ෝ
𝒚
Pretrained jointly
[This model has learned how to represent
entire sentences through pretraining]
What can we learn from reconstructing the input?
11
Stanford University is located in __________, California.
What can we learn from reconstructing the input?
12
I put ___ fork down on the table.
What can we learn from reconstructing the input?
13
The woman walked across the street,
checking for traffic over ___ shoulder.
What can we learn from reconstructing the input?
14
I went to the ocean to see the fish, turtles, seals, and _____.
What can we learn from reconstructing the input?
15
Overall, the value I got from the two hours watching
it was the sum total of the popcorn and the drink.
The movie was ___.
What can we learn from reconstructing the input?
16
Iroh went into the kitchen to make some tea.
Standing next to Iroh, Zuko pondered his destiny.
Zuko left the ______.
What can we learn from reconstructing the input?
17
I was thinking about the sequence that goes
1, 1, 2, 3, 5, 8, 13, 21, ____
The Transformer Encoder-Decoder [Vaswani et al., 2017]
Transformer
Encoder
Word
Embeddings
Position
Representations
+
Transformer
Encoder
[input sequence]
Transformer
Decoder
Word
Embeddings
Position
Representations
+
Transformer
Decoder
[output sequence]
[decoder attends
to encoder states]
Looking back at the whole model, zooming in on an Encoder block:
[predictions!]
18
The Transformer Encoder-Decoder [Vaswani et al., 2017]
Word
Embeddings
Position
Representations
+
Transformer
Encoder
[input sequence]
Transformer
Decoder
Word
Embeddings
Position
Representations
+
Transformer
Decoder
[output sequence]
[decoder attends
to encoder states]
Looking back at the whole model, zooming in on an Encoder block:
[predictions!]
Multi-Head Attention
Residual + LayerNorm
Feed-Forward
Residual + LayerNorm
19
The Transformer Encoder-Decoder [Vaswani et al., 2017]
Transformer
Encoder
Word
Embeddings
Position
Representations
+
Transformer
Encoder
[input sequence]
Word
Embeddings
Position
Representations
+
Transformer
Decoder
[output sequence]
Looking back at the whole model,
zooming in on a Decoder block:
[predictions!]
Residual + LayerNorm
Multi-Head Cross-Attention
Masked Multi-Head Self-Attention
Residual + LayerNorm
Feed-Forward
Residual + LayerNorm
20
The Transformer Encoder-Decoder [Vaswani et al., 2017]
Transformer
Encoder
Word
Embeddings
Position
Representations
+
Transformer
Encoder
[input sequence]
Word
Embeddings
Position
Representations
+
Transformer
Decoder
[output sequence]
The only new part is attention from decoder to encoder.
Like we saw last week!
[predictions!]
Residual + LayerNorm
Multi-Head Cross-Attention
Masked Multi-Head Self-Attention
Residual + LayerNorm
Feed-Forward
Residual + LayerNorm
21
Pretraining through language modeling [Dai and Le, 2015]
Recall the language modeling task:
• Model 𝑝𝜃 𝑤𝑡 𝑤1:𝑡−1), the probability
distribution over words given their past
contexts.
• There’s lots of data for this! (In English.)
Pretraining through language modeling:
• Train a neural network to perform language
modeling on a large amount of text.
• Save the network parameters.
22
Decoder
(Transformer, LSTM, ++ )
Iroh goes to make tasty tea
goes to make tasty tea END
The Pretraining / Finetuning Paradigm
Pretraining can improve NLP applications by serving as parameter initialization.
23
Decoder
(Transformer, LSTM, ++ )
Iroh goes to make tasty tea
goes to make tasty tea END
Step 1: Pretrain (on language modeling)
Lots of text; learn general things!
Decoder
(Transformer, LSTM, ++ )
☺/
Step 2: Finetune (on your task)
Not many labels; adapt to the task!
… the movie was …
Stochastic gradient descent and pretrain/finetune
Why should pretraining and finetuning help, from a “training neural nets” perspective?
• Consider, provides parameters ෠
𝜃 by approximating min
𝜃
ℒpretrain 𝜃 .
• (The pretraining loss.)
• Then, finetuning approximates min
𝜃
ℒfinetune 𝜃 , starting at ෠
𝜃.
• (The finetuning loss)
• The pretraining may matter because stochastic gradient descent sticks (relatively)
close to ෠
𝜃 during finetuning.
• So, maybe the finetuning local minima near ෠
𝜃 tend to generalize well!
• And/or, maybe the gradients of finetuning loss near ෠
𝜃 propagate nicely!
24
Lecture Plan
1. A brief note on subword modeling
2. Motivating model pretraining from word embeddings
3. Model pretraining three ways
1. Decoders
2. Encoders
3. Encoder-Decoders
4. Interlude: what do we think pretraining is teaching?
5. Very large models and in-context learning
25
Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.
26
Decoders
• Language models! What we’ve seen so far.
• Nice to generate from; can’t condition on future words
Encoders
• Gets bidirectional context – can condition on future!
• Wait, how do we pretrain them?
Encoder-
Decoders
• Good parts of decoders and encoders?
• What’s the best way to pretrain them?
Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.
27
Decoders
• Language models! What we’ve seen so far.
• Nice to generate from; can’t condition on future words
Encoders
• Gets bidirectional context – can condition on future!
• Wait, how do we pretrain them?
Encoder-
Decoders
• Good parts of decoders and encoders?
• What’s the best way to pretrain them?
ℎ1, … , ℎ𝑇
Pretraining decoders
When using language model pretrained decoders, we can ignore
that they were trained to model 𝑝 𝑤𝑡 𝑤1:𝑡−1).
28
We can finetune them by training a classifier
on the last word’s hidden state.
ℎ1, … , ℎ𝑇 = Decoder 𝑤1, … , 𝑤𝑇
𝑦 ∼ 𝐴𝑤𝑇 + 𝑏
Where 𝐴 and 𝑏 are randomly initialized and
specified by the downstream task.
Gradients backpropagate through the whole
network.
☺/
𝑤1, … , 𝑤𝑇
Linear 𝐴, 𝑏
[Note how the linear layer hasn’t been
pretrained and must be learned from scratch.]
Pretraining decoders
It’s natural to pretrain decoders as language models and then
use them as generators, finetuning their 𝑝𝜃 𝑤𝑡 𝑤1:𝑡−1)!
29
This is helpful in tasks where the output is a
sequence with a vocabulary like that at
pretraining time!
• Dialogue (context=dialogue history)
• Summarization (context=document)
ℎ1, … , ℎ𝑇 = Decoder 𝑤1, … , 𝑤𝑇
𝑤𝑡 ∼ 𝐴𝑤𝑡−1 + 𝑏
Where 𝐴, 𝑏 were pretrained in the language
model!
𝑤2 𝑤3 𝑤4 𝑤5 𝑤6
[Note how the linear layer has been pretrained.]
𝐴, 𝑏
ℎ1, … , ℎ𝑇
𝑤1 𝑤2 𝑤3 𝑤4 𝑤5
Generative Pretrained Transformer (GPT) [Radford et al., 2018]
2018’s GPT was a big success in pretraining a decoder!
• Transformer decoder with 12 layers.
• 768-dimensional hidden states, 3072-dimensional feed-forward hidden layers.
• Byte-pair encoding with 40,000 merges
• Trained on BooksCorpus: over 7000 unique books.
• Contains long spans of contiguous text, for learning long-distance dependencies.
• The acronym “GPT” never showed up in the original paper; it could stand for
“Generative PreTraining” or “Generative Pretrained Transformer”
30 [Devlin et al., 2018]
Generative Pretrained Transformer (GPT) [Radford et al., 2018]
How do we format inputs to our decoder for finetuning tasks?
Natural Language Inference: Label pairs of sentences as entailing/contradictory/neutral
Premise: The man is in the doorway
Hypothesis: The person is near the door
Radford et al., 2018 evaluate on natural language inference.
Here’s roughly how the input was formatted, as a sequence of tokens for the decoder.
[START] The man is in the doorway [DELIM] The person is near the door [EXTRACT]
The linear classifier is applied to the representation of the [EXTRACT] token.
31
entailment
Generative Pretrained Transformer (GPT) [Radford et al., 2018]
GPT results on various natural language inference datasets.
32
We mentioned how pretrained decoders can be used in their capacities as language models.
GPT-2, a larger version of GPT trained on more data, was shown to produce relatively
convincing samples of natural language.
Increasingly convincing generations (GPT2) [Radford et al., 2018]
Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.
34
Decoders
• Language models! What we’ve seen so far.
• Nice to generate from; can’t condition on future words
Encoders
• Gets bidirectional context – can condition on future!
• Wait, how do we pretrain them?
Encoder-
Decoders
• Good parts of decoders and encoders?
• What’s the best way to pretrain them?
ℎ1, … , ℎ𝑇
Pretraining encoders: what pretraining objective to use?
So far, we’ve looked at language model pretraining. But encoders get bidirectional
context, so we can’t do language modeling!
35
Idea: replace some fraction of words in the
input with a special [MASK] token; predict
these words.
ℎ1, … , ℎ𝑇 = Encoder 𝑤1, … , 𝑤𝑇
𝑦𝑖 ∼ 𝐴𝑤𝑖 + 𝑏
Only add loss terms from words that are
“masked out.” If ෤
𝑥 is the masked version of 𝑥,
we’re learning 𝑝𝜃(𝑥|෤
𝑥). Called Masked LM.
I [M] to the [M]
went store
𝐴, 𝑏
[Devlin et al., 2018]
BERT: Bidirectional Encoder Representations from Tranformers
Devlin et al., 2018 proposed the “Masked LM” objective and released the weights of a
pretrained Transformer, a model they labeled BERT.
36
Some more details about Masked LM for BERT:
• Predict a random 15% of (sub)word tokens.
• Replace input word with [MASK] 80% of the time
• Replace input word with a random token 10% of
the time
• Leave input word unchanged 10% of the time (but
still predict it!)
• Why? Doesn’t let the model get complacent and not
build strong representations of non-masked words.
(No masks are seen at fine-tuning time!)
[Predict these!]
I pizza to the [M]
went store
Transformer
Encoder
[Devlin et al., 2018]
to
[Masked]
[Replaced] [Not replaced]
BERT: Bidirectional Encoder Representations from Tranformers
37
• The pretraining input to BERT was two separate contiguous chunks of text:
• BERT was trained to predict whether one chunk follows the other or is randomly
sampled.
• Later work has argued this “next sentence prediction” is not necessary.
[Devlin et al., 2018, Liu et al., 2019]
BERT: Bidirectional Encoder Representations from Tranformers
Details about BERT
• Two models were released:
• BERT-base: 12 layers, 768-dim hidden states, 12 attention heads, 110 million params.
• BERT-large: 24 layers, 1024-dim hidden states, 16 attention heads, 340 million params.
• Trained on:
• BooksCorpus (800 million words)
• English Wikipedia (2,500 million words)
• Pretraining is expensive and impractical on a single GPU.
• BERT was pretrained with 64 TPU chips for a total of 4 days.
• (TPUs are special tensor operation acceleration hardware)
• Finetuning is practical and common on a single GPU
• “Pretrain once, finetune many times.”
38 [Devlin et al., 2018]
BERT: Bidirectional Encoder Representations from Tranformers
BERT was massively popular and hugely versatile; finetuning BERT led to new state-of-
the-art results on a broad range of tasks.
39
• QQP: Quora Question Pairs (detect paraphrase
questions)
• QNLI: natural language inference over question
answering data
• SST-2: sentiment analysis
• CoLA: corpus of linguistic acceptability (detect
whether sentences are grammatical.)
• STS-B: semantic textual similarity
• MRPC: microsoft paraphrase corpus
• RTE: a small natural language inference corpus
[Devlin et al., 2018]
Limitations of pretrained encoders
Those results looked great! Why not used pretrained encoders for everything?
40
If your task involves generating sequences, consider using a pretrained decoder; BERT and other
pretrained encoders don’t naturally lead to nice autoregressive (1-word-at-a-time) generation
methods.
Pretrained Encoder
Iroh goes to [MASK] tasty tea
make/brew/craft
Pretrained Decoder
Iroh goes to make tasty tea
goes to make tasty tea END
Extensions of BERT
You’ll see a lot of BERT variants like RoBERTa, SpanBERT, +++
41
Some generally accepted improvements to the BERT pretraining formula:
• RoBERTa: mainly just train BERT for longer and remove next sentence prediction!
• SpanBERT: masking contiguous spans of words makes a harder, more useful pretraining task
[Liu et al., 2019; Joshi et al., 2020]
BERT
[MASK] irr## esi## sti## [MASK] good
It’s
SpanBERT
bly
It’ [MASK] good
irr## esi## sti## bly
[MASK]
[MASK]
[MASK]
Extensions of BERT
A takeaway from the RoBERTa paper: more compute, more data can improve pretraining
even when not changing the underlying Transformer encoder.
42 [Liu et al., 2019; Joshi et al., 2020]
Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.
43
Decoders
• Language models! What we’ve seen so far.
• Nice to generate from; can’t condition on future words
Encoders
• Gets bidirectional context – can condition on future!
• Wait, how do we pretrain them?
Encoder-
Decoders
• Good parts of decoders and encoders?
• What’s the best way to pretrain them?
Pretraining encoder-decoders: what pretraining objective to use?
For encoder-decoders, we could do something like language modeling, but where a
prefix of every input is provided to the encoder and is not predicted.
44
ℎ1, … , ℎ𝑇 = Encoder 𝑤1, … , 𝑤𝑇
ℎ𝑇+1, … , ℎ2 = 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑤1, … , 𝑤𝑇, ℎ1, … , ℎ𝑇
𝑦𝑖 ∼ 𝐴𝑤𝑖 + 𝑏, 𝑖 > 𝑇
The encoder portion benefits from
bidirectional context; the decoder portion is
used to train the whole model through
language modeling.
[Raffel et al., 2018]
𝑤1, … , 𝑤𝑇
𝑤𝑇+1, … , 𝑤2𝑇
𝑤𝑇+2, … ,
Pretraining encoder-decoders: what pretraining objective to use?
What Raffel et al., 2018 found to work best was span corruption. Their model: T5.
45
Replace different-length spans from the input
with unique placeholders; decode out the
spans that were removed!
This is implemented in text
preprocessing: it’s still an objective
that looks like language modeling at
the decoder side.
Pretraining encoder-decoders: what pretraining objective to use?
Raffel et al., 2018 found encoder-decoders to work better than decoders for their tasks,
and span corruption (denoising) to work better than language modeling.
Pretraining encoder-decoders: what pretraining objective to use?
A fascinating property
of T5: it can be
finetuned to answer a
wide range of
questions, retrieving
knowledge from its
parameters.
NQ: Natural Questions
WQ: WebQuestions
TQA: Trivia QA
All “open-domain”
versions
[Raffel et al., 2018]
220 million params
770 million params
3 billion params
11 billion params
Outline
1. A brief note on subword modeling
2. Motivating model pretraining from word embeddings
3. Model pretraining three ways
1. Decoders
2. Encoders
3. Encoder-Decoders
4. Interlude: what do we think pretraining is teaching?
5. Very large models and in-context learning
48
What kinds of things does pretraining learn?
There’s increasing evidence that pretrained models learn a wide variety of things about
the statistical properties of language. Taking our examples from the start of class:
• Stanford University is located in __________, California. [Trivia]
• I put ___ fork down on the table. [syntax]
• The woman walked across the street, checking for traffic over ___ shoulder. [coreference]
• I went to the ocean to see the fish, turtles, seals, and _____. [lexical semantics/topic]
• Overall, the value I got from the two hours watching it was the sum total of the popcorn
and the drink. The movie was ___. [sentiment]
• Iroh went into the kitchen to make some tea. Standing next to Iroh, Zuko pondered his
destiny. Zuko left the ______. [some reasoning – this is harder]
• I was thinking about the sequence that goes 1, 1, 2, 3, 5, 8, 13, 21, ____ [some basic
arithmetic; they don’t learn the Fibonnaci sequence]
• Models also learn – and can exacerbate racism, sexism, all manner of bad biases.
• More on all this in the interpretability lecture!
49
Outline
1. A brief note on subword modeling
2. Motivating model pretraining from word embeddings
3. Model pretraining three ways
1. Decoders
2. Encoders
3. Encoder-Decoders
4. Interlude: what do we think pretraining is teaching?
5. Very large models and in-context learning
50
GPT-3, In-context learning, and very large models
So far, we’ve interacted with pretrained models in two ways:
• Sample from the distributions they define (maybe providing a prompt)
• Fine-tune them on a task we care about, and take their predictions.
Very large language models seem to perform some kind of learning without gradient
steps simply from examples you provide within their contexts.
GPT-3 is the canonical example of this. The largest T5 model had 11 billion parameters.
GPT-3 has 175 billion parameters.
51
GPT-3, In-context learning, and very large models
Very large language models seem to perform some kind of learning without gradient
steps simply from examples you provide within their contexts.
The in-context examples seem to specify the task to be performed, and the conditional
distribution mocks performing the task to a certain extent.
Input (prefix within a single Transformer decoder context):
“ thanks -> merci
hello -> bonjour
mint -> menthe
otter -> ”
Output (conditional generations):
loutre…”
52
GPT-3, In-context learning, and very large models
Very large language models seem to perform some kind of learning without gradient
steps simply from examples you provide within their contexts.
53
Parting remarks
These models are still not well-understood.
“Small” models like BERT have become general tools in a wide range of settings.
More on this in later lectures!
Assignment 5 out today! Tuesday’s and today’s lectures in its subject matter.
54

More Related Content

PDF
cs229_lecture_slides_selfsupervision_final.pdf
PPTX
Machine Translation Machine Translation Machine Translation
PPTX
Mattingly "AI & Prompt Design: Large Language Models"
PDF
Open vocabulary problem
PDF
NLP using transformers
PPT
Moore_slides.ppt
PDF
overview of natural language processing concepts
PPTX
Deep Learning for Natural Language Processing
cs229_lecture_slides_selfsupervision_final.pdf
Machine Translation Machine Translation Machine Translation
Mattingly "AI & Prompt Design: Large Language Models"
Open vocabulary problem
NLP using transformers
Moore_slides.ppt
overview of natural language processing concepts
Deep Learning for Natural Language Processing

Similar to Deep network notes.pdf (20)

PPTX
NLP Deep Dive - recurrent neural networks .pptx
PPTX
A Panorama of Natural Language Processing
PPTX
NLP WORDEMBEDDDING TECHINUES CBOW BOW.pptx
PDF
The NLP Muppets revolution!
PPTX
Pycon ke word vectors
PDF
Junki Matsuo - 2015 - Source Phrase Segmentation and Translation for Japanese...
PDF
Beyond the Symbols: A 30-minute Overview of NLP
PDF
Neural machine translation of rare words with subword units
PPTX
Deep Learning for Natural Language Processing
PPTX
Introduction to Neural Information Retrieval and Large Language Models
PDF
Python overview
PDF
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
PDF
AINL 2016: Nikolenko
PDF
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
PPTX
ورشة تضمين الكلمات في التعلم العميق Word embeddings workshop
PPTX
MixedLanguageProcessingTutorialEMNLP2019.pptx
PPTX
NLP.pptx
PPTX
KiwiPyCon 2014 - NLP with Python tutorial
PPTX
LLM24aug.pptxxz khi ong mat troi thuc dat me
PDF
NLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Deep Dive - recurrent neural networks .pptx
A Panorama of Natural Language Processing
NLP WORDEMBEDDDING TECHINUES CBOW BOW.pptx
The NLP Muppets revolution!
Pycon ke word vectors
Junki Matsuo - 2015 - Source Phrase Segmentation and Translation for Japanese...
Beyond the Symbols: A 30-minute Overview of NLP
Neural machine translation of rare words with subword units
Deep Learning for Natural Language Processing
Introduction to Neural Information Retrieval and Large Language Models
Python overview
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
AINL 2016: Nikolenko
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
ورشة تضمين الكلمات في التعلم العميق Word embeddings workshop
MixedLanguageProcessingTutorialEMNLP2019.pptx
NLP.pptx
KiwiPyCon 2014 - NLP with Python tutorial
LLM24aug.pptxxz khi ong mat troi thuc dat me
NLP Bootcamp 2018 : Representation Learning of text for NLP
Ad

More from Ramya Nellutla (17)

PDF
Module -6.pdf Machine Learning Types and examples
PDF
Module 5.pdf Machine Learning Types and examples
PPTX
Unit -3.pptx cloud Security unit -3 notes
PPT
MOBILE SECURITY -UNIT -II PPT IV- PROFESSIONAL ELEFCTIVE
PPTX
Unit-1Mobile Security Notes Mobile Communication
PPT
artificial Intelligence unit1 ppt (1).ppt
PDF
pentration testing.pdf
PPTX
Deep Learning.pptx
PDF
Unit-I PPT.pdf
PDF
- Social Engineering Unit- II Part- I.pdf
PPTX
Datamodels.pptx
PPT
Unit-3-Part-1 [Autosaved].ppt
PDF
E5-roughsets unit-V.pdf
PPTX
Unit-3.pptx
PDF
Unit-II -Soft Computing.pdf
PPT
SC01_IntroductionSC-Unit-I.ppt
PPTX
- Fuzzy Systems -II.pptx
Module -6.pdf Machine Learning Types and examples
Module 5.pdf Machine Learning Types and examples
Unit -3.pptx cloud Security unit -3 notes
MOBILE SECURITY -UNIT -II PPT IV- PROFESSIONAL ELEFCTIVE
Unit-1Mobile Security Notes Mobile Communication
artificial Intelligence unit1 ppt (1).ppt
pentration testing.pdf
Deep Learning.pptx
Unit-I PPT.pdf
- Social Engineering Unit- II Part- I.pdf
Datamodels.pptx
Unit-3-Part-1 [Autosaved].ppt
E5-roughsets unit-V.pdf
Unit-3.pptx
Unit-II -Soft Computing.pdf
SC01_IntroductionSC-Unit-I.ppt
- Fuzzy Systems -II.pptx
Ad

Recently uploaded (20)

PPTX
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
PPTX
Renaissance Architecture: A Journey from Faith to Humanism
PDF
Sports Quiz easy sports quiz sports quiz
PPTX
master seminar digital applications in india
PDF
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
PDF
Insiders guide to clinical Medicine.pdf
PDF
O5-L3 Freight Transport Ops (International) V1.pdf
PDF
RMMM.pdf make it easy to upload and study
PPTX
Final Presentation General Medicine 03-08-2024.pptx
PPTX
PPH.pptx obstetrics and gynecology in nursing
PDF
TR - Agricultural Crops Production NC III.pdf
PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
PPTX
Microbial diseases, their pathogenesis and prophylaxis
PPTX
Pharmacology of Heart Failure /Pharmacotherapy of CHF
PDF
01-Introduction-to-Information-Management.pdf
PPTX
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
PDF
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
PDF
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
PDF
Anesthesia in Laparoscopic Surgery in India
PDF
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
Renaissance Architecture: A Journey from Faith to Humanism
Sports Quiz easy sports quiz sports quiz
master seminar digital applications in india
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
Insiders guide to clinical Medicine.pdf
O5-L3 Freight Transport Ops (International) V1.pdf
RMMM.pdf make it easy to upload and study
Final Presentation General Medicine 03-08-2024.pptx
PPH.pptx obstetrics and gynecology in nursing
TR - Agricultural Crops Production NC III.pdf
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
Microbial diseases, their pathogenesis and prophylaxis
Pharmacology of Heart Failure /Pharmacotherapy of CHF
01-Introduction-to-Information-Management.pdf
IMMUNITY IMMUNITY refers to protection against infection, and the immune syst...
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
Anesthesia in Laparoscopic Surgery in India
Saundersa Comprehensive Review for the NCLEX-RN Examination.pdf

Deep network notes.pdf

  • 1. Natural Language Processing with Deep Learning CS224N/Ling284 John Hewitt Lecture 10: Pretraining
  • 2. Lecture Plan 1. A brief note on subword modeling 2. Motivating model pretraining from word embeddings 3. Model pretraining three ways 1. Decoders 2. Encoders 3. Encoder-Decoders 4. Interlude: what do we think pretraining is teaching? 5. Very large models and in-context learning Reminders: Assignment 5 is out today! It covers lecture 9 (Tuesday) and lecture 10 (Today)! It has ~pedagogically relevant math~ so get started! 2
  • 3. Word structure and subword models Let’s take a look at the assumptions we’ve made about a language’s vocabulary. We assume a fixed vocab of tens of thousands of words, built from the training set. All novel words seen at test time are mapped to a single UNK. word vocab mapping embedding hat → pizza (index) learn → tasty (index) taaaaasty → UNK (index) laern → UNK (index) Transformerify → UNK (index) 3 Common words Variations misspellings novel items
  • 4. Word structure and subword models Finite vocabulary assumptions make even less sense in many languages. • Many languages exhibit complex morphology, or word structure. • The effect is more word types, each occurring fewer times. 4 Example: Swahili verbs can have hundreds of conjugations, each encoding a wide variety of information. (Tense, mood, definiteness, negation, information about the object, ++) Here’s a small fraction of the conjugations for ambia – to tell. [Wiktionary]
  • 5. The byte-pair encoding algorithm Subword modeling in NLP encompasses a wide range of methods for reasoning about structure below the word level. (Parts of words, characters, bytes.) • The dominant modern paradigm is to learn a vocabulary of parts of words (subword tokens). • At training and testing time, each word is split into a sequence of known subwords. Byte-pair encoding is a simple, effective strategy for defining a subword vocabulary. 1. Start with a vocabulary containing only characters and an “end-of-word” symbol. 2. Using a corpus of text, find the most common adjacent characters “a,b”; add “ab” as a subword. 3. Replace instances of the character pair with the new subword; repeat until desired vocab size. Originally used in NLP for machine translation; now a similar method (WordPiece) is used in pretrained models. 5 [Sennrich et al., 2016, Wu et al., 2016]
  • 6. Word structure and subword models Common words end up being a part of the subword vocabulary, while rarer words are split into (sometimes intuitive, sometimes not) components. In the worst case, words are split into as many subwords as they have characters. word vocab mapping embedding hat → hat learn → learn taaaaasty → taa## aaa## sty laern → la## ern## Transformerify → Transformer## ify 6 Common words Variations misspellings novel items
  • 7. Outline 1. A brief note on subword modeling 2. Motivating model pretraining from word embeddings 3. Model pretraining three ways 1. Decoders 2. Encoders 3. Encoder-Decoders 4. Interlude: what do we think pretraining is teaching? 5. Very large models and in-context learning 7
  • 8. Motivating word meaning and context Recall the adage we mentioned at the beginning of the course: “You shall know a word by the company it keeps” (J. R. Firth 1957: 11) This quote is a summary of distributional semantics, and motivated word2vec. But: “… the complete meaning of a word is always contextual, and no study of meaning apart from a complete context can be taken seriously.” (J. R. Firth 1935) Consider I record the record: the two instances of record mean different things. 8 [Thanks to Yoav Goldberg on Twitter for pointing out the 1935 Firth quote.]
  • 9. Where we were: pretrained word embeddings Circa 2017: • Start with pretrained word embeddings (no context!) • Learn how to incorporate context in an LSTM or Transformer while training on the task. Some issues to think about: • The training data we have for our downstream task (like question answering) must be sufficient to teach all contextual aspects of language. • Most of the parameters in our network are randomly initialized! 9 … the movie was … ෝ 𝒚 Not pretrained pretrained (word embeddings) [Recall, movie gets the same word embedding, no matter what sentence it shows up in]
  • 10. Where we’re going: pretraining whole models In modern NLP: • All (or almost all) parameters in NLP networks are initialized via pretraining. • Pretraining methods hide parts of the input from the model, and train the model to reconstruct those parts. • This has been exceptionally effective at building strong: • representations of language • parameter initializations for strong NLP models. • Probability distributions over language that we can sample from 10 … the movie was … ෝ 𝒚 Pretrained jointly [This model has learned how to represent entire sentences through pretraining]
  • 11. What can we learn from reconstructing the input? 11 Stanford University is located in __________, California.
  • 12. What can we learn from reconstructing the input? 12 I put ___ fork down on the table.
  • 13. What can we learn from reconstructing the input? 13 The woman walked across the street, checking for traffic over ___ shoulder.
  • 14. What can we learn from reconstructing the input? 14 I went to the ocean to see the fish, turtles, seals, and _____.
  • 15. What can we learn from reconstructing the input? 15 Overall, the value I got from the two hours watching it was the sum total of the popcorn and the drink. The movie was ___.
  • 16. What can we learn from reconstructing the input? 16 Iroh went into the kitchen to make some tea. Standing next to Iroh, Zuko pondered his destiny. Zuko left the ______.
  • 17. What can we learn from reconstructing the input? 17 I was thinking about the sequence that goes 1, 1, 2, 3, 5, 8, 13, 21, ____
  • 18. The Transformer Encoder-Decoder [Vaswani et al., 2017] Transformer Encoder Word Embeddings Position Representations + Transformer Encoder [input sequence] Transformer Decoder Word Embeddings Position Representations + Transformer Decoder [output sequence] [decoder attends to encoder states] Looking back at the whole model, zooming in on an Encoder block: [predictions!] 18
  • 19. The Transformer Encoder-Decoder [Vaswani et al., 2017] Word Embeddings Position Representations + Transformer Encoder [input sequence] Transformer Decoder Word Embeddings Position Representations + Transformer Decoder [output sequence] [decoder attends to encoder states] Looking back at the whole model, zooming in on an Encoder block: [predictions!] Multi-Head Attention Residual + LayerNorm Feed-Forward Residual + LayerNorm 19
  • 20. The Transformer Encoder-Decoder [Vaswani et al., 2017] Transformer Encoder Word Embeddings Position Representations + Transformer Encoder [input sequence] Word Embeddings Position Representations + Transformer Decoder [output sequence] Looking back at the whole model, zooming in on a Decoder block: [predictions!] Residual + LayerNorm Multi-Head Cross-Attention Masked Multi-Head Self-Attention Residual + LayerNorm Feed-Forward Residual + LayerNorm 20
  • 21. The Transformer Encoder-Decoder [Vaswani et al., 2017] Transformer Encoder Word Embeddings Position Representations + Transformer Encoder [input sequence] Word Embeddings Position Representations + Transformer Decoder [output sequence] The only new part is attention from decoder to encoder. Like we saw last week! [predictions!] Residual + LayerNorm Multi-Head Cross-Attention Masked Multi-Head Self-Attention Residual + LayerNorm Feed-Forward Residual + LayerNorm 21
  • 22. Pretraining through language modeling [Dai and Le, 2015] Recall the language modeling task: • Model 𝑝𝜃 𝑤𝑡 𝑤1:𝑡−1), the probability distribution over words given their past contexts. • There’s lots of data for this! (In English.) Pretraining through language modeling: • Train a neural network to perform language modeling on a large amount of text. • Save the network parameters. 22 Decoder (Transformer, LSTM, ++ ) Iroh goes to make tasty tea goes to make tasty tea END
  • 23. The Pretraining / Finetuning Paradigm Pretraining can improve NLP applications by serving as parameter initialization. 23 Decoder (Transformer, LSTM, ++ ) Iroh goes to make tasty tea goes to make tasty tea END Step 1: Pretrain (on language modeling) Lots of text; learn general things! Decoder (Transformer, LSTM, ++ ) ☺/ Step 2: Finetune (on your task) Not many labels; adapt to the task! … the movie was …
  • 24. Stochastic gradient descent and pretrain/finetune Why should pretraining and finetuning help, from a “training neural nets” perspective? • Consider, provides parameters ෠ 𝜃 by approximating min 𝜃 ℒpretrain 𝜃 . • (The pretraining loss.) • Then, finetuning approximates min 𝜃 ℒfinetune 𝜃 , starting at ෠ 𝜃. • (The finetuning loss) • The pretraining may matter because stochastic gradient descent sticks (relatively) close to ෠ 𝜃 during finetuning. • So, maybe the finetuning local minima near ෠ 𝜃 tend to generalize well! • And/or, maybe the gradients of finetuning loss near ෠ 𝜃 propagate nicely! 24
  • 25. Lecture Plan 1. A brief note on subword modeling 2. Motivating model pretraining from word embeddings 3. Model pretraining three ways 1. Decoders 2. Encoders 3. Encoder-Decoders 4. Interlude: what do we think pretraining is teaching? 5. Very large models and in-context learning 25
  • 26. Pretraining for three types of architectures The neural architecture influences the type of pretraining, and natural use cases. 26 Decoders • Language models! What we’ve seen so far. • Nice to generate from; can’t condition on future words Encoders • Gets bidirectional context – can condition on future! • Wait, how do we pretrain them? Encoder- Decoders • Good parts of decoders and encoders? • What’s the best way to pretrain them?
  • 27. Pretraining for three types of architectures The neural architecture influences the type of pretraining, and natural use cases. 27 Decoders • Language models! What we’ve seen so far. • Nice to generate from; can’t condition on future words Encoders • Gets bidirectional context – can condition on future! • Wait, how do we pretrain them? Encoder- Decoders • Good parts of decoders and encoders? • What’s the best way to pretrain them?
  • 28. ℎ1, … , ℎ𝑇 Pretraining decoders When using language model pretrained decoders, we can ignore that they were trained to model 𝑝 𝑤𝑡 𝑤1:𝑡−1). 28 We can finetune them by training a classifier on the last word’s hidden state. ℎ1, … , ℎ𝑇 = Decoder 𝑤1, … , 𝑤𝑇 𝑦 ∼ 𝐴𝑤𝑇 + 𝑏 Where 𝐴 and 𝑏 are randomly initialized and specified by the downstream task. Gradients backpropagate through the whole network. ☺/ 𝑤1, … , 𝑤𝑇 Linear 𝐴, 𝑏 [Note how the linear layer hasn’t been pretrained and must be learned from scratch.]
  • 29. Pretraining decoders It’s natural to pretrain decoders as language models and then use them as generators, finetuning their 𝑝𝜃 𝑤𝑡 𝑤1:𝑡−1)! 29 This is helpful in tasks where the output is a sequence with a vocabulary like that at pretraining time! • Dialogue (context=dialogue history) • Summarization (context=document) ℎ1, … , ℎ𝑇 = Decoder 𝑤1, … , 𝑤𝑇 𝑤𝑡 ∼ 𝐴𝑤𝑡−1 + 𝑏 Where 𝐴, 𝑏 were pretrained in the language model! 𝑤2 𝑤3 𝑤4 𝑤5 𝑤6 [Note how the linear layer has been pretrained.] 𝐴, 𝑏 ℎ1, … , ℎ𝑇 𝑤1 𝑤2 𝑤3 𝑤4 𝑤5
  • 30. Generative Pretrained Transformer (GPT) [Radford et al., 2018] 2018’s GPT was a big success in pretraining a decoder! • Transformer decoder with 12 layers. • 768-dimensional hidden states, 3072-dimensional feed-forward hidden layers. • Byte-pair encoding with 40,000 merges • Trained on BooksCorpus: over 7000 unique books. • Contains long spans of contiguous text, for learning long-distance dependencies. • The acronym “GPT” never showed up in the original paper; it could stand for “Generative PreTraining” or “Generative Pretrained Transformer” 30 [Devlin et al., 2018]
  • 31. Generative Pretrained Transformer (GPT) [Radford et al., 2018] How do we format inputs to our decoder for finetuning tasks? Natural Language Inference: Label pairs of sentences as entailing/contradictory/neutral Premise: The man is in the doorway Hypothesis: The person is near the door Radford et al., 2018 evaluate on natural language inference. Here’s roughly how the input was formatted, as a sequence of tokens for the decoder. [START] The man is in the doorway [DELIM] The person is near the door [EXTRACT] The linear classifier is applied to the representation of the [EXTRACT] token. 31 entailment
  • 32. Generative Pretrained Transformer (GPT) [Radford et al., 2018] GPT results on various natural language inference datasets. 32
  • 33. We mentioned how pretrained decoders can be used in their capacities as language models. GPT-2, a larger version of GPT trained on more data, was shown to produce relatively convincing samples of natural language. Increasingly convincing generations (GPT2) [Radford et al., 2018]
  • 34. Pretraining for three types of architectures The neural architecture influences the type of pretraining, and natural use cases. 34 Decoders • Language models! What we’ve seen so far. • Nice to generate from; can’t condition on future words Encoders • Gets bidirectional context – can condition on future! • Wait, how do we pretrain them? Encoder- Decoders • Good parts of decoders and encoders? • What’s the best way to pretrain them?
  • 35. ℎ1, … , ℎ𝑇 Pretraining encoders: what pretraining objective to use? So far, we’ve looked at language model pretraining. But encoders get bidirectional context, so we can’t do language modeling! 35 Idea: replace some fraction of words in the input with a special [MASK] token; predict these words. ℎ1, … , ℎ𝑇 = Encoder 𝑤1, … , 𝑤𝑇 𝑦𝑖 ∼ 𝐴𝑤𝑖 + 𝑏 Only add loss terms from words that are “masked out.” If ෤ 𝑥 is the masked version of 𝑥, we’re learning 𝑝𝜃(𝑥|෤ 𝑥). Called Masked LM. I [M] to the [M] went store 𝐴, 𝑏 [Devlin et al., 2018]
  • 36. BERT: Bidirectional Encoder Representations from Tranformers Devlin et al., 2018 proposed the “Masked LM” objective and released the weights of a pretrained Transformer, a model they labeled BERT. 36 Some more details about Masked LM for BERT: • Predict a random 15% of (sub)word tokens. • Replace input word with [MASK] 80% of the time • Replace input word with a random token 10% of the time • Leave input word unchanged 10% of the time (but still predict it!) • Why? Doesn’t let the model get complacent and not build strong representations of non-masked words. (No masks are seen at fine-tuning time!) [Predict these!] I pizza to the [M] went store Transformer Encoder [Devlin et al., 2018] to [Masked] [Replaced] [Not replaced]
  • 37. BERT: Bidirectional Encoder Representations from Tranformers 37 • The pretraining input to BERT was two separate contiguous chunks of text: • BERT was trained to predict whether one chunk follows the other or is randomly sampled. • Later work has argued this “next sentence prediction” is not necessary. [Devlin et al., 2018, Liu et al., 2019]
  • 38. BERT: Bidirectional Encoder Representations from Tranformers Details about BERT • Two models were released: • BERT-base: 12 layers, 768-dim hidden states, 12 attention heads, 110 million params. • BERT-large: 24 layers, 1024-dim hidden states, 16 attention heads, 340 million params. • Trained on: • BooksCorpus (800 million words) • English Wikipedia (2,500 million words) • Pretraining is expensive and impractical on a single GPU. • BERT was pretrained with 64 TPU chips for a total of 4 days. • (TPUs are special tensor operation acceleration hardware) • Finetuning is practical and common on a single GPU • “Pretrain once, finetune many times.” 38 [Devlin et al., 2018]
  • 39. BERT: Bidirectional Encoder Representations from Tranformers BERT was massively popular and hugely versatile; finetuning BERT led to new state-of- the-art results on a broad range of tasks. 39 • QQP: Quora Question Pairs (detect paraphrase questions) • QNLI: natural language inference over question answering data • SST-2: sentiment analysis • CoLA: corpus of linguistic acceptability (detect whether sentences are grammatical.) • STS-B: semantic textual similarity • MRPC: microsoft paraphrase corpus • RTE: a small natural language inference corpus [Devlin et al., 2018]
  • 40. Limitations of pretrained encoders Those results looked great! Why not used pretrained encoders for everything? 40 If your task involves generating sequences, consider using a pretrained decoder; BERT and other pretrained encoders don’t naturally lead to nice autoregressive (1-word-at-a-time) generation methods. Pretrained Encoder Iroh goes to [MASK] tasty tea make/brew/craft Pretrained Decoder Iroh goes to make tasty tea goes to make tasty tea END
  • 41. Extensions of BERT You’ll see a lot of BERT variants like RoBERTa, SpanBERT, +++ 41 Some generally accepted improvements to the BERT pretraining formula: • RoBERTa: mainly just train BERT for longer and remove next sentence prediction! • SpanBERT: masking contiguous spans of words makes a harder, more useful pretraining task [Liu et al., 2019; Joshi et al., 2020] BERT [MASK] irr## esi## sti## [MASK] good It’s SpanBERT bly It’ [MASK] good irr## esi## sti## bly [MASK] [MASK] [MASK]
  • 42. Extensions of BERT A takeaway from the RoBERTa paper: more compute, more data can improve pretraining even when not changing the underlying Transformer encoder. 42 [Liu et al., 2019; Joshi et al., 2020]
  • 43. Pretraining for three types of architectures The neural architecture influences the type of pretraining, and natural use cases. 43 Decoders • Language models! What we’ve seen so far. • Nice to generate from; can’t condition on future words Encoders • Gets bidirectional context – can condition on future! • Wait, how do we pretrain them? Encoder- Decoders • Good parts of decoders and encoders? • What’s the best way to pretrain them?
  • 44. Pretraining encoder-decoders: what pretraining objective to use? For encoder-decoders, we could do something like language modeling, but where a prefix of every input is provided to the encoder and is not predicted. 44 ℎ1, … , ℎ𝑇 = Encoder 𝑤1, … , 𝑤𝑇 ℎ𝑇+1, … , ℎ2 = 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑤1, … , 𝑤𝑇, ℎ1, … , ℎ𝑇 𝑦𝑖 ∼ 𝐴𝑤𝑖 + 𝑏, 𝑖 > 𝑇 The encoder portion benefits from bidirectional context; the decoder portion is used to train the whole model through language modeling. [Raffel et al., 2018] 𝑤1, … , 𝑤𝑇 𝑤𝑇+1, … , 𝑤2𝑇 𝑤𝑇+2, … ,
  • 45. Pretraining encoder-decoders: what pretraining objective to use? What Raffel et al., 2018 found to work best was span corruption. Their model: T5. 45 Replace different-length spans from the input with unique placeholders; decode out the spans that were removed! This is implemented in text preprocessing: it’s still an objective that looks like language modeling at the decoder side.
  • 46. Pretraining encoder-decoders: what pretraining objective to use? Raffel et al., 2018 found encoder-decoders to work better than decoders for their tasks, and span corruption (denoising) to work better than language modeling.
  • 47. Pretraining encoder-decoders: what pretraining objective to use? A fascinating property of T5: it can be finetuned to answer a wide range of questions, retrieving knowledge from its parameters. NQ: Natural Questions WQ: WebQuestions TQA: Trivia QA All “open-domain” versions [Raffel et al., 2018] 220 million params 770 million params 3 billion params 11 billion params
  • 48. Outline 1. A brief note on subword modeling 2. Motivating model pretraining from word embeddings 3. Model pretraining three ways 1. Decoders 2. Encoders 3. Encoder-Decoders 4. Interlude: what do we think pretraining is teaching? 5. Very large models and in-context learning 48
  • 49. What kinds of things does pretraining learn? There’s increasing evidence that pretrained models learn a wide variety of things about the statistical properties of language. Taking our examples from the start of class: • Stanford University is located in __________, California. [Trivia] • I put ___ fork down on the table. [syntax] • The woman walked across the street, checking for traffic over ___ shoulder. [coreference] • I went to the ocean to see the fish, turtles, seals, and _____. [lexical semantics/topic] • Overall, the value I got from the two hours watching it was the sum total of the popcorn and the drink. The movie was ___. [sentiment] • Iroh went into the kitchen to make some tea. Standing next to Iroh, Zuko pondered his destiny. Zuko left the ______. [some reasoning – this is harder] • I was thinking about the sequence that goes 1, 1, 2, 3, 5, 8, 13, 21, ____ [some basic arithmetic; they don’t learn the Fibonnaci sequence] • Models also learn – and can exacerbate racism, sexism, all manner of bad biases. • More on all this in the interpretability lecture! 49
  • 50. Outline 1. A brief note on subword modeling 2. Motivating model pretraining from word embeddings 3. Model pretraining three ways 1. Decoders 2. Encoders 3. Encoder-Decoders 4. Interlude: what do we think pretraining is teaching? 5. Very large models and in-context learning 50
  • 51. GPT-3, In-context learning, and very large models So far, we’ve interacted with pretrained models in two ways: • Sample from the distributions they define (maybe providing a prompt) • Fine-tune them on a task we care about, and take their predictions. Very large language models seem to perform some kind of learning without gradient steps simply from examples you provide within their contexts. GPT-3 is the canonical example of this. The largest T5 model had 11 billion parameters. GPT-3 has 175 billion parameters. 51
  • 52. GPT-3, In-context learning, and very large models Very large language models seem to perform some kind of learning without gradient steps simply from examples you provide within their contexts. The in-context examples seem to specify the task to be performed, and the conditional distribution mocks performing the task to a certain extent. Input (prefix within a single Transformer decoder context): “ thanks -> merci hello -> bonjour mint -> menthe otter -> ” Output (conditional generations): loutre…” 52
  • 53. GPT-3, In-context learning, and very large models Very large language models seem to perform some kind of learning without gradient steps simply from examples you provide within their contexts. 53
  • 54. Parting remarks These models are still not well-understood. “Small” models like BERT have become general tools in a wide range of settings. More on this in later lectures! Assignment 5 out today! Tuesday’s and today’s lectures in its subject matter. 54