![•
• Latent Variable Model [Arora+,2016]
•
• [Mimno
and Thompson, 2017]
• Paraphrase Model [Gittens+, 2017]
•
• Zipf’s
4
https://ja.wikipedia.org/wiki/](https://image.slidesharecdn.com/pub20191102aclmoura-191109143218/85/Towards-Understanding-Linear-Word-Analogies-4-320.jpg)
![• SGNS PMI [Levy & Goldberg 2014]
• SGNS
PMI
• NS
•
• k>0 SGNS negative sample
• GloVe log
⃗x
⃗y c
6
⟨ ⃗x , ⃗y c⟩ = log Xx,y − bx − by
⟨ ⃗x , ⃗y c⟩ = PMI(x, y) − log k](https://image.slidesharecdn.com/pub20191102aclmoura-191109143218/85/Towards-Understanding-Linear-Word-Analogies-6-320.jpg)
![• [Mikolov+2013]
•
•
8
https://github.com/tmikolov/word2vec/blob/master/questions-words.txt
capital-world](https://image.slidesharecdn.com/pub20191102aclmoura-191109143218/85/Towards-Understanding-Linear-Word-Analogies-8-320.jpg)
![csPMI
•
• -csPMI
• [Pennungton+’14][Arora+’16]
• (16) (2) 2
11
(csPMI(x, y) ∈ {−∞,0])
p(w|king)
p(w|queen)
≈
p(w|king)
p(w|queen)
(w ∈ Vocab) ⃗king − ⃗man + ⃗woman ≈ ⃗queen](https://image.slidesharecdn.com/pub20191102aclmoura-191109143218/85/Towards-Understanding-Linear-Word-Analogies-11-320.jpg)
[論文紹介] Towards Understanding Linear Word Analogies
- 1. Towards Understanding Linear Word Analogies ACL LT , 2019/11/02,
- 2. • Towards Understanding Linear Word Analogies • Kawin Ethayarajh, David Duvenaud, Graeme Hirst • University of Toronto • ACL 2019 • • SGNS GloVe (1) (2) csPMI 2
- 3. • SGNS GloVe • ⃗king − ⃗man + ⃗woman ≈ ⃗queen 3
- 4. • • Latent Variable Model [Arora+,2016] • • [Mimno and Thompson, 2017] • Paraphrase Model [Gittens+, 2017] • • Zipf’s 4 https://ja.wikipedia.org/wiki/
- 5. • 5
- 6. • SGNS PMI [Levy & Goldberg 2014] • SGNS PMI • NS • • k>0 SGNS negative sample • GloVe log ⃗x ⃗y c 6 ⟨ ⃗x , ⃗y c⟩ = log Xx,y − bx − by ⟨ ⃗x , ⃗y c⟩ = PMI(x, y) − log k
- 7. Co-occurrence Shifted PMI Theorem • • S (x, y) csPMI(x, y) • S (x, y), (a, b) csPMI(a, x) = csPMI(b, y) • a, b, x, y • king - man + woman = queen • csPMI(king, queen) = csPMI(man, woman) • csPMI(man, king) = csPMI(woman, queen) • man, woman, king, queen 7 x a b y csPMI(x, y) = PMI(x, y) + log p(x, y)
- 9. csPMI 9 ⃗wc = λ ⃗w p(w) ∝ p(w, w) α ∈ ℝ− γ′, λ ∈ ℝ, α ∈ ℝ−
- 10. csPMI • • • null csPMI • csPMI • ” ” cf:IDF • : (x,y) =(the, apple) z = “the apple” • p(the) > p(apple) • csMPI(“the apple”, “the”) < csPMI(“the apple”, “apple”) ⃗z = ⃗x + ⃗y ∅ ⃗0 csPMI(x, z) = csPMI(∅, y) ⇒ csPMI(x, z) = log p(y) + δ ⃗z ⃗x ⃗y p(y) p(x) > p(y) ⟺ csPMI(z, y) > csPMI(z, x) 10 δ
- 11. csPMI • • -csPMI • [Pennungton+’14][Arora+’16] • (16) (2) 2 11 (csPMI(x, y) ∈ {−∞,0]) p(w|king) p(w|queen) ≈ p(w|king) p(w|queen) (w ∈ Vocab) ⃗king − ⃗man + ⃗woman ≈ ⃗queen
- 12. • • csPMI • csPMI • csPMI 12
- 13. csPMI 13
- 15. csPMI • csPMI → csPMI • • Fig2 • (x,y) • OK • csPMI • csPMI cf. Table1 currency εx,y = Mx,y − < ⃗x , ⃗y c > 15 ⟨ ⃗x , ⃗y c⟩ = PMI(x, y) − log k ⟨ ⃗x , ⃗y c⟩ ≈ PMI(x, y) − log k ⟨ ⃗x , ⃗y c⟩ ≠ PMI(x, y) − log k
- 16. • csPMI • csPMI= • csPMI 16