Understanding GloVe

Understanding
GloVe
(Global Vectors for Word Representation)
-
Slides written by Park JeeHyun
27 DEC 2017

Contents
1. Introduction
2. GloVe model
3. GloVe cost function
4. Experiments & Results
5. References

1. Introduction
• The statistics of word occurrences in a corpus is the primary
source of information available to all unsupervised methods
for learning word representations.
• Although many such methods now exist, the question still
remains as to how meaning is generated from these
statistics, and how the resulting word vectors might
represent that meaning.

1. Introduction
• Recent methods for learning vector space representations
of words have succeeded in capturing fine-grained
semantic and syntactic regularities using vector arithmetic,
• But the origin of these regularities has remained opaque.

2. GloVe model
• Combines the advantages of the two major model families
in the literature:
• global matrix factorization and,
• local context window methods
• Our model efficiently leverages statistical information by
training only on the nonzero elements in a word-word co-
occurrence matrix, rather than on the entire sparse matrix or
on individual context windows in a large corpus.

• Deriving the Cost Function
• The above argument suggests that the appropriate starting point for word vector
learning should be with ratios of co-occurrence probabilities
rather than the probabilities themselves.

• Establish some notations
• let the matrix of word-word co-occurrence counts be denoted by 𝑋,
• whose entries 𝑋𝑖𝑗 tabulate the number of times word 𝑗
occurs in the context of word 𝑖
• let 𝑋𝑖 = 𝑘 𝑋𝑖𝑘 be the number of times any word
appears in the context of word 𝑖
• let 𝑃𝑖𝑗 = 𝑃 𝑖 𝑗 = 𝑋𝑖𝑗 𝑋𝑖 be the probability that word j
appear in the context of word 𝑖

• set a function F that represents ratios of co-occurrence probabilities rather than t
he probabilities themselves
• we would like F to encode the information present the ratio 𝑃𝑖𝑘 𝑃𝑗𝑘 in the
word vector space. Since vector spaces are inherently linear structures,
the most natural way to do this is with vector differences.
 Note that 𝑤𝑖 and 𝑤 𝑘 are v
ectors from different
vector-spaces

• We note that the arguments of F in Eqn. (2) are vectors while the right-hand sid
e is a scalar.
• While F could be taken to be a complicated function parameterized by,
e.g., a neural network, doing so would obfuscate the linear structure we are tryin
g to capture.
• To avoid this issue, we can first take the dot product of the arguments,
which prevents F from mixing the vector dimensions in undesirable ways.

• for word-word co-occurrence matrices,
the distinction between a word and a context word is arbitrary
and that we are free to exchange the two roles.
• the symmetry can be restored in two steps.
• First, we require that 𝐹 be a homomorphism between the groups
(ℝ,+) and (ℝ >0, ×), i.e.,
• which, by Eqn. (3), is solved by,

3. GloVe cost function : homomorphism

Homo (same)
+
Morph (shape)
3. GloVe cost function : homomorphism

• The solution to Eqn. (4) is 𝐹 = 𝑒𝑥𝑝 or,
• Next, we note that Eqn. (6) would exhibit the exchange symmetry
if not for the 𝐥𝐨𝐠(𝑿𝒊) on the right-hand side.
• However, this term is independent of 𝑘 so it can be absorbed into
a bias 𝑏𝑖 for 𝑤𝑖.
• Finally, adding an additional bias 𝑏𝑖 for 𝑤𝑖 restores the symmetry.

3. GloVe cost function : weighting function

4. Experiments & Results
• Experiments
• Word analogy
• Word similarity
• Named entity recognition (NER)
• Result

4. Experiments : analogy task
• The word analogy task consists of questions like,
“a is to b as c is to ___?”
• The semantic questions, like
“Athens is to Greece as Berlin is to ___?”.
• The syntactic questions like,
“dance is to dancing as fly is to ___?”

4. Experiments : similarity task
• A similarity score is obtained from the word vectors by first
normalizing each feature across the vocabulary and then
calculating the cosine similarity.
• We compute Spearman’s rank correlation coefficient
between this score and the human judgments.

4. Experiments : similarity task

4. Experiments : NER task
• The CoNLL-2003 English benchmark dataset for NER is a
collection of documents from Reuters newswire articles,
annotated with four entity types:
• Person
• Location
• Organization
• Miscellaneous
• We train models on CoNLL-03 training data on test on three
datasets:
1) ConLL-03 testing data
2) ACE Phase 2 (2001-02) and ACE-2003 data
3) MUC7 Formal Run test set.

4. Result
• GloVe, is a new global log-bilinear regression model for the
unsupervised learning of word representations that
outperforms other models on word analogy, word similarity,
and named entity recognition tasks.

5. References
• GloVe
• https://nlp.stanford.edu/projects/glove/
• Stanford NLP lecture
• http://web.stanford.edu/class/cs224n/
• https://www.youtube.com/watch?v=ASn7ExxLZws&t=43s
• Socratica’s video about homomorphism
• https://www.youtube.com/watch?v=cYzp5IWqCsg

Understanding GloVe

More Related Content

What's hot (20)

Similar to Understanding GloVe (20)

More from JEE HYUN PARK (8)

Recently uploaded (20)

Understanding GloVe