「知識」のDeep Learning

Deep Learning
1 Preferred Infrastructure
(@unnonouno)
2015/06/04 PFI

NLP 1YANS
!  YANS 19
!  140
! 
! 
!  YANS 13
! 
!  "

Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
!  RNN
! 
! 
!  LSTM

Long Short-Term Memory (LSTM)
! 
! 
1
!

! 
! 
argmaxy P(y|x) = argmaxy P(x|y) P(y)

-‐‑‒ [ 15]
!  Pydata Tokyo

!  EMNLP2014 Bordes Weston
1 part 2 [Bordes&Weston14]

Q: Deep Learning
Representation Learning
-‐‑‒
A: Deep

推論を導けるような知識の表現、およびその方法
を開発する人工知能研究の領域である。

1
!  2 t t 3
!  t t
!  RDF
(x, r, y)
x yr

! 
!  t t
!  t t
!  t t

-‐‑‒
! 
-‐‑‒
NLP

!  {(xi, ri, yi)}:
!  x, y: t t
!  r: t t
!  x, y
!  r

! 
! 
1
argmax ∑i f(xi, ri, yi)

Distance model (Structured Embedding) [Bordes
+11]
!  e
!  Rleft, Rright
! 
!  f
f(x, r, y) = || Rleft(r) e(x) – Rright(r) e(y) ||1

TransE model [Brodes+13]
!  r r
! 
f(x, r, y) = || e(x) + r – e(y) ||2
2

TransE model
1 TransE
!  TransM:
!  TransH:

TransM model [Fan+14]
!  r
!  wr r x, y
f(x, r, y) = wr|| e(x) + r – e(y) ||2
2

TransH model [Wang+14]
!  TransE

Bilinear model
!  r
! 
f(x, r, y) = e(x)T Wr e(y)

Neural Tensor Network (NTN) [Socher+13]
f(x, r, y) = ur tanh(e(x)Wre(y)
+ V1
r e(x) + V2
r e(y) + br)
!  r
!

[Yang+15]
!  2
! 
Bilinear

Link prediction
!  t t 1
!  QA
!  -‐‑‒
!  t t
(e1, r, e2) (e1, r, ? )

TransH
TransE
[Bordes&Weston14]

[Weston+13]
!  x y r
! 
[Bordes&Weston14]

[Weston+13]
! 
! 
-‐‑‒
[Bordes&Weston14]

Link prediction 1QA
!  Link prediction
!

QA [Bordes+14]
! 
!  q t
f(q) g(t)
!  f(q), g(t) q, t

Memory networks:
[Weston+15]
!  I: I(x)
!  G: mi = G(mi, I(x), m)
!  O: o = O(I(x), m)
!  R: r = R(o)

-‐‑‒
! 
!  -‐‑‒
!  -‐‑‒
!  -‐‑‒
! 
!  factoid
!  -‐‑‒
!

! 
! 
! 
! 
! 
! 
! 
!

1/4
!  [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S.
Khudanpur.
Recurrent neural network based language model.
Interspeech 2010.
!  [ 15] .
.
2015.
!  [ 15] .
NLP Introduction based on Project Next NLP.
PyData.Tokyo Meetup #5, 2015.
!  [Bordes&Weston14] A. Bordes, J. Weston.
Embedding Methods for Natural Language Processing.
EMNLP2014 tutorial.
!  [ 15] .
.
JSAI 2015 .

2/4
!  [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio.
Learning structured embeddings of knowledge bases.
AAAI2011.
!  [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston,
O. Yakhnenko.
Translating Embeddings for Modeling Multi-relational Data.
NIPS 2013.
!  [Fan+14] M. Fan, Q. Shou, E. Chang, T. F. Zheng.
Transition-based Knowledge Graph Embedding with Relational
Mapping Properties.
PACLIC 2014.
!  [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen.
Knowledge Graph Embedding by Translating on Hyperplanes.
AAAI 2014.

3/4
!  [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng.
Reasoning With Neural Tensor Networks for Knowledge Base
Completion.
NIPS 2013.
!  [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng.
Embedding Entities and Relations for Learning and Inferenece
in Knowledge Bases.
ICLR 2015.
!  [Nickel+11] M. Nickel, V. Tresp, H. P. Kriegel.
A Three-Way Model for Collective Learning on Multi-Relational
Data.
ICML 2011.

4/4
!  [Bordes+14] A. Bordes, J. Weston, N. Usunier.
Open Question Answering with Weakly Supervised
Embedding Models.
ECML/PKDD 2014.
!  [Weston+15] J. Weston, S. Chopra, A. Bordes.
Memory Networks.
ICLR 2015.
!  [Peng&Yao15] B. Peng, K. Yao.
Recurrent Neural Networks with External Memory
for Language Understanding.
arXiv:1506.00195, 2015.

「知識」のDeep Learning

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to 「知識」のDeep Learning (20)

More from Yuya Unno (18)

Recently uploaded (20)

「知識」のDeep Learning