![[EMNLP] What is GloVe? Part II
An introduction to unsupervised learning of word embeddings from
co-occurrence matrices.
Brendan Whitaker
May 25, 2018 · 5 min read
In this article, we’ll discuss one of the newer methods of creating vector space models
of word semantics, more commonly known as word embeddings. The original paper by
J. Pennington, R. Socher, and C. Manning is available here:
http://www.aclweb.org/anthology/D14-1162.This method combines elements from
the two main word embedding models which existed when GloVe, short for “Global
Vectors [for word representation]” was proposed: global matrix factorization and local
context window methods. In Part I, we compared these two different approaches. Now
we’ll give an explanation of the GloVe embedding generation algorithm and how it
improves on these previous methods.](https://image.slidesharecdn.com/emnlpwhatisglovepartii-towardsdatascience-200228052050/85/Emnlp-what-is-glo-ve-part-ii-towards-data-science-1-320.jpg)
![Note that the w’s are real-valued word vectors. Now since we want to encode
information about the ratios between two words, the authors suggest using vector
differences as inputs to our function. Then we have the following:
Now we’re getting closer to something that could work, but the final function for the
GloVe model will be considerably more complex to accurately reflect certain desirable
symmetries, since distinction between the words i and j should be invariant under
commutation of the inputs. The authors also design a weighting scheme for co-
occurrences to reflect the relationship between frequency and semantic relevance. But
since I’m trying to keep these summary articles to around 800 words, we’ll cover all
that in Part III!
[EMNLP] What is GloVe? Part III
An introduction to unsupervised learning of word
embeddings from co-occurrence matrices.
towardsdatascience.com
Please check out the source paper!
Page 1 of 12](https://image.slidesharecdn.com/emnlpwhatisglovepartii-towardsdatascience-200228052050/85/Emnlp-what-is-glo-ve-part-ii-towards-data-science-4-320.jpg)
[Emnlp] what is glo ve part ii - towards data science
- 1. [EMNLP] What is GloVe? Part II An introduction to unsupervised learning of word embeddings from co-occurrence matrices. Brendan Whitaker May 25, 2018 · 5 min read In this article, we’ll discuss one of the newer methods of creating vector space models of word semantics, more commonly known as word embeddings. The original paper by J. Pennington, R. Socher, and C. Manning is available here: http://www.aclweb.org/anthology/D14-1162.This method combines elements from the two main word embedding models which existed when GloVe, short for “Global Vectors [for word representation]” was proposed: global matrix factorization and local context window methods. In Part I, we compared these two different approaches. Now we’ll give an explanation of the GloVe embedding generation algorithm and how it improves on these previous methods.
- 2. . . . Co-occurrence probabilities. Recall from the Part I of this series that term-term frequency matrices encode how often terms appear in the context of one another by enumerating each unique token in the corpus along both of the axes of a large 2-dimensional matrix. Performing matrix factorization gives us a low rank approximation of the whole of the data contained in the original matrix. However, as we’ll explain in a moment, the authors of GloVe discovered via empirical methods that instead of learning the raw co-occurrence probabilities, it may make more sense to learn ratios of these co-occurrence probabilities, which seem to better discriminate subtleties in term-term relevance. To illustrate this, we borrow an example from their paper: suppose we wish to study the relationship between two words, i = ice and j = steam. We’ll do this by examining the co-occurrence probabilities of these words with various “probe” words. We define the co-occurrence probability of an arbitrary word i with an arbitrary word j to be the probability that word j appears in the context of word i. This is represented by the equation and definitions below. Note X_i is defined as the number of times any word appears in the context of word i, so it’s defined as the sum over all words k of the number of times word k occurs in the context of word i. So if we choose a probe word k = solid which is closely related to i = ice but not to j = steam, we expect the ratio P_{ik}/P_{jk} of co-occurrence probabilities to be large, since solid should, in theory, appear in the context of ice more often than it would appear in the context of steam, since ice is a solid and steam is not.Conversely, for a choice of k = gas, we would expect the same ratio to be small,
- 3. since steam is more closely related to gas than ice is. Then we also have words like water, which are closely related to both ice and steam, but not more to one than the other. And we also have words like fashion which are not closely related to either of the words in question. For both water and fashion, we expect our ratio to be close to 1 since there shouldn’t be any bias to one of ice or steam. Now it is important to note that since we are trying to determine information about the relationship between the words ice and steam, water doesn’t give us a lot of useful information. For discriminative purposes, it doesn’t give us a good idea of how “far apart” steam is from ice, and the information that steam, ice, and water are all related is already captured in the discriminative information between ice and water, and steam and water. Words that don’t help us distinguish between i and j are referred to as noise points, and it is the use of the ratio between co-occurrence probabilities helps filter out these noise points. This is well-illustrated by the real data for these example points, which we print here from Table 1 in the GloVe paper. source So let’s take a step back for a moment and recall the overall structure of the problem. We want to take data from the corpus in the form of global statistics and learn a function that gives us information about the relationship between any two words in said corpus, given only the words themselves. Now the authors have discovered that ratios of co-occurrence probabilities are a good source of this information, so it would be nice if our function mapped from the space of two words to compare as well as a context word to the space of co-occurrence probability ratios. So let the function our model is learning be given by F. A naive interpretation of the desired model is given by the authors as:
- 4. Note that the w’s are real-valued word vectors. Now since we want to encode information about the ratios between two words, the authors suggest using vector differences as inputs to our function. Then we have the following: Now we’re getting closer to something that could work, but the final function for the GloVe model will be considerably more complex to accurately reflect certain desirable symmetries, since distinction between the words i and j should be invariant under commutation of the inputs. The authors also design a weighting scheme for co- occurrences to reflect the relationship between frequency and semantic relevance. But since I’m trying to keep these summary articles to around 800 words, we’ll cover all that in Part III! [EMNLP] What is GloVe? Part III An introduction to unsupervised learning of word embeddings from co-occurrence matrices. towardsdatascience.com Please check out the source paper! Page 1 of 12
- 5. Machine Learning Arti cial Intelligence Language Tech Linguistics About Help Legal GloVe: Global Vectors for Word Representation Jeffrey Pennington, Richard Socher, Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA 94305jpennin@stanford.edu, richard@socher.org, manning@stanford.edu Abstract Recent methods for learning vector spacerepresentations of words have succeededin capturing fine-grained semantic andsyntactic regularities using vector arith- metic, but the origin of these regularitieshas remained opaque. We analyze andmake explicit the model properties neededfor such regularities to emerge in wordvectors. The result is a new global log- bilinear regression model that combinesthe advantages of the two major modelfamilies in the literature: global matrix factorization and local context windowmethods. Our model efficiently leveragesstatistical information by training only onthe nonzero elements in a word-word co- occurrence matrix, rather than on the en- tire sparse matrix or on individual contextwindows in a large corpus. The model pro- duces a vector space with meaningful sub- structure, as evidenced by its performanceof 75% on a recent word analogy task. Italso outperforms related models on simi- larity tasks and named entity recognition. 1 Introduction Semantic vector space models of language repre- sent each word with a real-valued vector. Thesevectors can be used as features in a variety of ap- plications, such as information retrieval (Manninget al., 2008), document classification (Sebastiani,2002), question answering (Tellex et al., 2003),named entity recognition (Turian et al., 2010), andparsing (Socher et al., 2013).Most word vector methods rely on the distanceor angle between pairs of word vectors as the pri- mary method for evaluating the intrinsic qualityof such a set of word representations. Recently,Mikolov et al. (2013c) introduced a new evalua- tion scheme based on word analogies that probes the finer structure of the word vector space by ex- amining not the scalar distance between word vec- tors, but rather their various dimensions of dif- ference. For example, the analogy “king is toqueen as man is to woman” should be encodedin the vector space by the vector equation king −queen = man − woman. This evaluation schemefavors models that produce dimensions of mean- ing, thereby capturing the multi- clustering idea ofdistributed representations (Bengio, 2009).The two main model families for learning wordvectors are: 1) global matrix factorization meth- ods, such as latent semantic analysis (LSA) (Deer- wester et al., 1990) and 2) local context windowmethods, such as the skip-gram model of Mikolovet al. (2013c). Currently, both families suffer sig- nificant drawbacks. While methods like LSA ef- ficiently leverage statistical information, they dorelatively poorly on the word analogy task, indi- cating a sub-optimal vector space structure. Meth- ods like skip-gram may do better on the analogytask, but they poorly utilize the statistics of the cor- pus since they train on separate local context win- dows instead of on global co-occurrence counts. In this work, we analyze the model propertiesnecessary to produce linear directions of meaningand argue that global log-bilinear regression mod- els are appropriate for doing so. We propose a spe- cific weighted least squares model that trains onglobal word-word co-occurrence counts and thusmakes efficient use of statistics. The model pro- duces a word vector space with meaningful sub- structure, as evidenced by its state-of-the-art per- formance of 75% accuracy on the word analogydataset. We also demonstrate that our methodsoutperform other current methods on several wordsimilarity tasks, and also on a common named en- tity recognition (NER) benchmark.We provide the source code for the model aswell as trained word vectors at http://nlp. stanford.edu/projects/glove/. Page 2 of 12 Page 1 / 12