![Co-occurrence matrix
◎ Let’s see window based co-occurrence matrix
◎ Example Corpus :
○ I like deep learning.
○ I like NLP.
○ I enjoy flying.
◎ Total vocabulary size(|V|) = 8
◎ Vector(“I”) = [0, 2, 1, 0, 0, 0, 0, 0]
◎ Vector(“like”) = [2, 0, 0, 1, 0, 1, 0 , 0]
4](https://image.slidesharecdn.com/efficientestimationofwordrepresentationsinvectorspace2013-210503204205/85/Efficient-estimation-of-word-representations-in-vector-space-2013-4-320.jpg)
![What
◎ For example – sentence = ” Word Embedding's are Word
converted into numbers ”
◎ A dictionary may be the list of all unique words in the sentence.
◎ So, a dictionary may look like –
[‘Word’,’Embedding's’,’are’,’Converted’,’into’,’numbers’]
◎ The vector representation of “numbers” according to the above
dictionary is [0,0,0,0,0,1] and of converted is[0,0,0,1,0,0].
6](https://image.slidesharecdn.com/efficientestimationofwordrepresentationsinvectorspace2013-210503204205/85/Efficient-estimation-of-word-representations-in-vector-space-2013-6-320.jpg)
Efficient estimation of word representations in vector space (2013)
- 1. Efficient Estimation of Word Representations in Vector Space • Tomas Mikolov • Kai Chen • Greg Corrado • Jeffrey Dean
- 2. “ (...) the meaning of a word is its use in the language. —Wittgenstein, Ludwig, Philosophical Investigations – 1953 2
- 3. Vector Space Model ◎ Traditional Vector Space Model (Information Retrieval): ○ documents and queries represented in a vector space ○ where the dimensions are the words 3
- 4. Co-occurrence matrix ◎ Let’s see window based co-occurrence matrix ◎ Example Corpus : ○ I like deep learning. ○ I like NLP. ○ I enjoy flying. ◎ Total vocabulary size(|V|) = 8 ◎ Vector(“I”) = [0, 2, 1, 0, 0, 0, 0, 0] ◎ Vector(“like”) = [2, 0, 0, 1, 0, 1, 0 , 0] 4
- 5. What ◎ A two layer neural network to generate word embedding's given a text corpus. ◎ Word Embedding's --- Mapping of words in a vector space ◎ So that similar words are mapped to nearby points 5
- 6. What ◎ For example – sentence = ” Word Embedding's are Word converted into numbers ” ◎ A dictionary may be the list of all unique words in the sentence. ◎ So, a dictionary may look like – [‘Word’,’Embedding's’,’are’,’Converted’,’into’,’numbers’] ◎ The vector representation of “numbers” according to the above dictionary is [0,0,0,0,0,1] and of converted is[0,0,0,1,0,0]. 6
- 7. Why ◎ Preserves relationship between words ◎ Deals with addition of new words in the vocabulary ◎ Better results in lots of deep learning application 7
- 9. CBOW ◎ Predict the target word from the context ◎ order of words in the history does not influence the projection ◎ faster & more appropriate for larger corpora 9
- 10. Skip Gram ◎ Predict the context words from target ◎ maximize classification of a word based on another word in the same sentence ◎ better word vectors for frequent words, but slower to train 10
- 11. A sliding window example 11
- 14. Advantages ◎ It scales ○ Train on billion word corpora ○ In limited time ○ Possibility of parallel training ◎ Pre-trained word embedding's trained by one can be used by others ○ For entirely different tasks ◎ Incremental training ○ Train on one piece of data, save results, continue training later on ◎ There is a Python module for it: ○ Gensim word2vec 14
- 15. Disadvantages ◎ Inability to handle unknown or OOV words. ... ◎ No shared representations at sub-word levels. ... ◎ Scaling to new languages requires new embedding matrices. ... ◎ Cannot be used to initialize state-of-the-art architectures. 15