企業における自然言語処理技術利用の最先端
78 likes15,944 views
1. This document discusses the history and recent developments in natural language processing and deep learning. It covers seminal NLP papers from the 1990s through 2000s and the rise of neural network approaches for NLP from 2003 onward. 2. Recent years have seen increased research and investment in deep learning, with many large companies establishing AI labs in 2012-2014 to focus on neural network techniques. 3. The document outlines some popular deep learning architectures for NLP tasks, including neural language models, word2vec, sequence-to-sequence learning, and memory networks. It also introduces the Chainer deep learning framework for Python.
![!
! 1993: [Brown+93]
! 1996: [Berger+96]
! 2001: [Lafferty+01]
! t
! 2003: Latent Dirichlet Allocation [Blei+03]
! 2006: Pitman-Yor language model [Teh06]
!
! 2006: [Clarke+06][Riedel+06]
! 2010: [Koo+10][Rush+10]
!
! 2003: Neural language model [Bengio+03]
! 2010: Recurrent Neural Network [Mikolov+10]
! 2013: Skipgram Model (word2vec) [Mikolov+13]](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-34-320.jpg)
![Neural Network Language Model (NNLM) [Bengio
+03]
! N
NN
! N-1](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-43-320.jpg)
![Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
! t-1
t
! NNLM N
!
! http://rnnlm.org](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-44-320.jpg)
![Skip-gram (word2vec) [Mikolov+13]
!
!
1CBOW
! Analogical reasoning
! 1word2vec](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-45-320.jpg)
![(MV-RNN) [Socher
+12]
!
!
f
! Root](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-46-320.jpg)
![Sequence-to-sequence learning [Sutskever+14]
!
!](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-47-320.jpg)
![Show and Tell [Vinyals+15]
!
! CNN RNN
CNN
CNN
RNN](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-48-320.jpg)
![Memory networks [Weston+15][Sukhbaatar+15]
!
!](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-49-320.jpg)
![(1/5)
! [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A.
Della Pietra, Robert L. Mercer.
The mathematics of statistical machine translation: parameter
estimation. Computational Linguistics Vol. 19 (2), 1993.
! [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A.
Della Pietra.
A Maximum Entropy Approach to Natural Language
Processing. Computational Linguistics, Vol. 22 (1), 1996.
! [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N.
Pereira.
Conditional Random Fields: Probabilistic Models for
Segmenting and Labeling Sequence Data. ICML2001.](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-61-320.jpg)
![(2/5)
! [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan.
Latent Dirichlet Allocation. JMLR Vol. 3, 2003.
! [Teh06] Yee Whye Teh.
A Hierarchical Bayesian Language Model based on Pitman-Yor
Processes. ACL 2006.
! [Clarke+06] James Clarke, Mirella Lapata.
Constraint-Based Sentence Compression: An Integer
Programming Approach. COLING/ACL 2006.
! [Riedel+06] Sebastian Riedel, James Clarke.
Incremental Integer Linear Programming for Non-projective
Dependency Parsing. COLING/ACL 2006.](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-62-320.jpg)
![(3/5)
! [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi
Jaakkola, David Sontag.
Dual Decomposition for Parsing with Non-Projective Head
Automata. EMNLP 2010.
! [Rush+10] Alexander M. Rush, David Sontag, Michael Collins,
Tommi Jaakkola.
On Dual Decomposition and Linear Programming Relaxations
for Natural Language Processing. EMNLP 2010.
! [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent,
Christian Jauvin.
A Neural Probabilistic Language Model. JMLR, 2003.](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-63-320.jpg)
![(4/5)
! [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan
"Honza" Cernocky, Sanjeev Khudanpur.
Recurrent neural network based language model.
Interspeech, 2010.
! [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean.
Efficient Estimation of Word Representations in Vector Space. CoRR,
2013.
! [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning,
Andrew Y. Ng.
Semantic Compositionality through Recursive Matrix-Vector Spaces.
EMNLP2012.
! [Sutskever+14] I. Sutskever, O. Vinyals, Q. V. Le.
Sequence to Sequence Learning with Neural Networks.
NIPS 2014.](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-64-320.jpg)
![(5/5)
! [Vinyals+15] O. Vinyals, A. Toshev, S. Bengio, D.
Erhan.
Show and Tell: A Neural Image Caption Generator.
arXiv:1411.4555, 2014.
! [Weston+15] J. Weston, S. Chopra, A. Bordes.
Memory Networks.
ICLR 2015
! [Sukhbaatar+15] S. Sukhbaatar, A. Szlam, J. Weston,
R. Fergus.
End-To-End Memory Networks.
arXiv:1503.08895, 2015.](https://image.slidesharecdn.com/20150619tosuken-150619160139-lva1-app6892/85/-65-320.jpg)
企業における自然言語処理技術利用の最先端
- 2. ! -2008 ! ! 2008-2011 t 1 ! t ! 2011- 1 t ! t ! ! Jubatus NLP 12014- t 12015,
- 3. NLP 1YANS ! YANS 19 ! ! -‐‑‒19/3 9/5 ! YANS 13 ! ! "
- 4. 1 ! !
- 7. 3 1. 2. t 3.
- 8. 1.
- 9. ! ! ---- ---- ! Web EC t Web ! 1CGM ---- ---- ! SNS1Twitter Facebook ! 1Line ! t
- 10. ! ! ! ! ! ! t WWW ! Blog ! SNS
- 12. ! EC t ! ! ! ! ! ! SNS ! ! t
- 13. ! EC t ! 11000 ! ! ! ! ! SNS ! !
- 16. ! ! ! ! ! ! 1 1 SNS ! t
- 17. ! ! ! ! ! ! 1
- 18. 2. t
- 20. ! !
- 21. t t : Siri https://www.apple.com/jp/ios/siri/ http://v-assist.yahoo.co.jp/ https://www.nttdocomo.co.jp/service/information/shabette_concier/
- 23. 11/2 ! ! 1970 t PC ! ! ! 2000 ! t
- 24. 12/2 ! ! RSS 2000 ! ! 1 ! 2000 !
- 25. ! ! t ! ! ! !
- 27. t
- 28. t ! !
- 29. ! ! <12:00 :25
- 30. t ! ! ! ! ! ! t
- 31. t ! t ! ! t ! t ! ! t
- 32. 3.
- 34. ! ! 1993: [Brown+93] ! 1996: [Berger+96] ! 2001: [Lafferty+01] ! t ! 2003: Latent Dirichlet Allocation [Blei+03] ! 2006: Pitman-Yor language model [Teh06] ! ! 2006: [Clarke+06][Riedel+06] ! 2010: [Koo+10][Rush+10] ! ! 2003: Neural language model [Bengio+03] ! 2010: Recurrent Neural Network [Mikolov+10] ! 2013: Skipgram Model (word2vec) [Mikolov+13]
- 35. ! ! 90 2010 !
- 36. ! !
- 37. 1. ! 2011: 30% # 20% ! 2012: 26% # 16% http://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdf
- 38. 2. ! 2012/3: Google Hinton DNNresearch ! 2012/4: Baidu Institute of Deep Learning ! 2012/8, 10: Yahoo! IQ Engines LookFlow ! 2012/12: Facebook AI Lab LeCun ! 2014/1: Google DeepMind ! 2014/5: Andrew Ng Baidu ! 2014/8: IBM SyNAPSE
- 39. 3. ! 2014/1 ! 2014/10 -‐‑‒ ! 2015/06
- 40. 4. t ! ! Sequence-to-sequence (2014, Google) ! ! Deep Q-Networks (2013, DeepMind) ! ! Variational Auto-Encoder (2014, UvA) ! Stochastic Backprop (2014, DeepMind) ! ! Neural Turing Machines (2014, DeepMind) ! ! Memory Networks (2014, Facebook) ! ! Show and Tell (2015, Google)
- 41. ! ! ! ! ! ! !
- 42. ! t ! ! ! 1 ! !
- 43. Neural Network Language Model (NNLM) [Bengio +03] ! N NN ! N-1
- 44. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] ! t-1 t ! NNLM N ! ! http://rnnlm.org
- 45. Skip-gram (word2vec) [Mikolov+13] ! ! 1CBOW ! Analogical reasoning ! 1word2vec
- 48. Show and Tell [Vinyals+15] ! ! CNN RNN CNN CNN RNN
- 51. ! libsvm, liblinear ! ! JUMAN, Chasen, MeCab ! ! Moses (GIZA++) ! ! Stanford CoreNLP ! ! word2vec ! Skipgram ! Theano, Caffe, Torch ! t !
- 52. Chainer v1.0.0 2015/06/09 Powerful Flexible Intuitive
- 53. Chainer ! Chainer ! Python h(f(x) + g(x)) x f g h+
- 56. t
- 57. PFI 2010 Xappy12012 ! 2012 Gunosy ! ! ! t
- 58. Xappy t ! ! Blog ! Twitter Facebook SNS ! ! 4G ! t ! ! etc. ! GUI t
- 59. ! ! ! ! t ! PC t ! ! ! ! 10 !
- 61. (1/5) ! [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics Vol. 19 (2), 1993. ! [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A. Della Pietra. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, Vol. 22 (1), 1996. ! [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. ICML2001.
- 62. (2/5) ! [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR Vol. 3, 2003. ! [Teh06] Yee Whye Teh. A Hierarchical Bayesian Language Model based on Pitman-Yor Processes. ACL 2006. ! [Clarke+06] James Clarke, Mirella Lapata. Constraint-Based Sentence Compression: An Integer Programming Approach. COLING/ACL 2006. ! [Riedel+06] Sebastian Riedel, James Clarke. Incremental Integer Linear Programming for Non-projective Dependency Parsing. COLING/ACL 2006.
- 63. (3/5) ! [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, David Sontag. Dual Decomposition for Parsing with Non-Projective Head Automata. EMNLP 2010. ! [Rush+10] Alexander M. Rush, David Sontag, Michael Collins, Tommi Jaakkola. On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing. EMNLP 2010. ! [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003.
- 64. (4/5) ! [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. ! [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013. ! [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix-Vector Spaces. EMNLP2012. ! [Sutskever+14] I. Sutskever, O. Vinyals, Q. V. Le. Sequence to Sequence Learning with Neural Networks. NIPS 2014.
- 65. (5/5) ! [Vinyals+15] O. Vinyals, A. Toshev, S. Bengio, D. Erhan. Show and Tell: A Neural Image Caption Generator. arXiv:1411.4555, 2014. ! [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015 ! [Sukhbaatar+15] S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus. End-To-End Memory Networks. arXiv:1503.08895, 2015.