Introduction to word embeddings with Python
4 likes3,976 views
This document discusses word embeddings and various techniques for generating them, including word2vec and GloVe models. Word2vec uses neural networks to learn vector representations of words from large amounts of text by predicting surrounding words. It has two architectures: CBOW and skip-gram. GloVe learns word vectors by leveraging global word-word co-occurrence statistics from a corpus, which helps represent rare words. The document provides examples of using word embeddings and discusses how they can be applied to tasks like generating hashtags and user interests.
![credits: [x]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/85/Introduction-to-word-embeddings-with-Python-15-320.jpg)
![here comes word2vec
Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al: [paper]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/85/Introduction-to-word-embeddings-with-Python-24-320.jpg)
![word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling
Word-Embedding Method, Goldberg et al, 2014 [arxiv]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/85/Introduction-to-word-embeddings-with-Python-38-320.jpg)
![deep walk:
DeepWalk: Online Learning of Social
Representations [link]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/85/Introduction-to-word-embeddings-with-Python-67-320.jpg)
![predicting hashtags
interesting read: #TAGSPACE: Semantic
Embeddings from Hashtags [link]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/85/Introduction-to-word-embeddings-with-Python-69-320.jpg)
Introduction to word embeddings with Python
- 1. Introduction to word embeddings Pavel Kalaidin @facultyofwonder Moscow Data Fest, September, 12th, 2015
- 5. лойс
- 6. годно, лойс лойс за песню из принципа не поставлю лойс взаимные лойсы лойс, если согласен What is the meaning of лойс?
- 7. годно, лойс лойс за песню из принципа не поставлю лойс взаимные лойсы лойс, если согласен What is the meaning of лойс?
- 8. кек
- 9. кек, что ли? кек))))))) ну ты кек What is the meaning of кек?
- 10. кек, что ли? кек))))))) ну ты кек What is the meaning of кек?
- 12. simple and flexible platform for understanding text and probably not messing up
- 13. one-hot encoding? 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
- 14. co-occurrence matrix recall: word-document co-occurrence matrix for LSA
- 15. credits: [x]
- 16. from entire document to window (length 5-10)
- 17. still seems suboptimal -> big, sparse, etc.
- 18. lower dimensions, we want dense vectors (say, 25-1000)
- 19. How?
- 22. lots of memory?
- 23. idea: directly learn low- dimensional vectors
- 24. here comes word2vec Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al: [paper]
- 25. idea: instead of capturing co- occurrence counts predict surrounding words
- 26. Two models: C-BOW predicting the word given its context skip-gram predicting the context given a word Explained in great detail here, so we’ll skip it for now Also see: word2vec Parameter Learning Explained, Rong, paper
- 28. CBOW: several times faster than skip-gram, slightly better accuracy for the frequent words Skip-Gram: works well with small amount of data, represents well rare words or phrases
- 29. Examples?
- 36. Wwoman - Wman = Wqueen - Wking classic example
- 38. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method, Goldberg et al, 2014 [arxiv]
- 39. all done with gensim: github.com/piskvorky/gensim/
- 40. ...failing to take advantage of the vast amount of repetition in the data
- 41. so back to co-occurrences
- 42. GloVe for Global Vectors Pennington et al, 2014: nlp.stanford. edu/pubs/glove.pdf
- 43. Ratios seem to cancel noise
- 44. The gist: model ratios with vectors
- 45. The model
- 49. recall:
- 51. Restoring symmetry, part 2
- 52. Least squares problem it is now
- 53. SGD->AdaGrad
- 54. ok, Python code
- 56. two sets of vectors input and context + bias average/sum/drop
- 57. complexity |V|2
- 65. Abusing models
- 67. deep walk: DeepWalk: Online Learning of Social Representations [link]
- 68. user interests Paragraph vectors: cs.stanford. edu/~quocle/paragraph_vector.pdf
- 69. predicting hashtags interesting read: #TAGSPACE: Semantic Embeddings from Hashtags [link]
- 70. RusVectōrēs: distributional semantic models for Russian: ling.go.mail. ru/dsm/en/
- 72. corpus matters
- 73. building block for bigger models ╰(*´︶`*)╯
- 74. </slides>