大規模データ時代に求められる自然言語処理 -言語情報から世界を捉える-
48 likes8,117 views
1. The document discusses the history and recent developments in natural language processing and deep learning. It provides an overview of seminal NLP papers from the 1990s to 2010s and deep learning architectures from 2003 to present. 2. Key deep learning models discussed include neural language models, word2vec, convolutional neural networks, and LSTMs. The document also notes the increasing interest and research in deep learning starting in 2012 by tech companies like Google, Facebook and Baidu. 3. Application examples mentioned include search engines, conversational agents, social media and news summarization tools.
![!
! 1993: [Brown+93]
! 1996: [Berger+96]
! 2001: [Lafferty+01]
!
! 2003: Latent Dirichlet Allocation [Blei+03]
! 2006: Pitman-Yor language model [Teh06]
!
! 2006: [Clarke+06][Riedel+06]
! 2010: [Koo+10][Rush+10]
!
! 2003: Neural language model [Bengio+03]
! 2010: Recurrent Neural Network [Mikolov+10]
! 2012: Skipgram Model (word2vec) [Mikolov+13]](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-43-320.jpg)
![Neural Network Language Model (NNLM) [Bengio
+03]
! N
NN
! N-1](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-51-320.jpg)
![Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
! t-1
t
! NNLM N
!
! http://rnnlm.org](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-52-320.jpg)
![Skip-gram (word2vec) [Mikolov+13]
!
!
CBOW
! Analogical reasoning
! word2vec](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-53-320.jpg)
![(MV-RNN) [Socher
+12]
!
!
f
! Root](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-54-320.jpg)
![Dynamic Convolutional Neural Network
[Kalchbrenner+14]
! CNN
!](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-55-320.jpg)
![(1/4)
! [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A.
Della Pietra, Robert L. Mercer.
The mathematics of statistical machine translation: parameter
estimation. Computational Linguistics Vol. 19 (2), 1993.
! [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A.
Della Pietra.
A Maximum Entropy Approach to Natural Language
Processing. Computational Linguistics, Vol. 22 (1), 1996.
! [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N.
Pereira.
Conditional Random Fields: Probabilistic Models for
Segmenting and Labeling Sequence Data. ICML2001.](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-93-320.jpg)
![(2/4)
! [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan.
Latent Dirichlet Allocation. JMLR Vol. 3, 2003.
! [Teh06] Yee Whye Teh.
A Hierarchical Bayesian Language Model based on Pitman-Yor
Processes. ACL 2006.
! [Clarke+06] James Clarke, Mirella Lapata.
Constraint-Based Sentence Compression: An Integer
Programming Approach. COLING/ACL 2006.
! [Riedel+06] Sebastian Riedel, James Clarke.
Incremental Integer Linear Programming for Non-projective
Dependency Parsing. COLING/ACL 2006.](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-94-320.jpg)
![(3/4)
! [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi
Jaakkola, David Sontag.
Dual Decomposition for Parsing with Non-Projective Head
Automata. EMNLP 2010.
! [Rush+10] Alexander M. Rush, David Sontag, Michael Collins,
Tommi Jaakkola.
On Dual Decomposition and Linear Programming Relaxations
for Natural Language Processing. EMNLP 2010.
! [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent,
Christian Jauvin.
A Neural Probabilistic Language Model. JMLR, 2003.](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-95-320.jpg)
![(4/4)
! [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan
"Honza" Cernocky, Sanjeev Khudanpur.
Recurrent neural network based language model.
Interspeech, 2010.
! [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean.
Efficient Estimation of Word Representations in Vector Space. CoRR,
2013.
! [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning,
Andrew Y. Ng.
Semantic Compositionality through Recursive Matrix-Vector Spaces.
EMNLP2012.
! [Kalchbrenner+14] Nal Kalchbrenner, Edward Grefenstette, Phil
Blunsom.
A Convolutional Neural Network for Modelling Sentences. ACL2014.](https://image.slidesharecdn.com/20150115tohoku-150117010048-conversion-gate01/85/-96-320.jpg)
大規模データ時代に求められる自然言語処理 -言語情報から世界を捉える-
- 1. - -
- 2. ! : 2006 3 ! : ! : 40 ! :
- 3. ! TopCoder’s Red Coder : 3 ! ICPC : 7 ! ICFP ! : 5 ! Publications: ACL, SIGKDD, NAACL, ALENEX, SODA, FOCS, STOC, DCC, SDM, ICDM, SPIRE, ESA, ICPR, MLJ, JMLR, NIPS, ECML/PKDD, Bioinformatics, Nucleic Acids Research ! Ex. SONY, IBM Research, Yahoo! Japan, Sun, Accenture, Microsoft, mixi, FAST Search&Transfer, TOPPAN Printing, NOMURA Security, GREE, KUMON, Google interns
- 4. ! ! ! ! !
- 5. ! -2008 ! ! 2008-2011 ! ! 2011- ! ! NLP 2014-
- 6. NLP YANS ! YANS 9 ! 40 ! ! ! YANS 3 ! ! "
- 8. ! !
- 9. 3 1. 2. 3.
- 10. 1.
- 11. ! ! ! ! ! ! WWW ! Blog ! SNS
- 13. ! ! ! ! ! ! !
- 15. ! ! ! ! ! !
- 18. ! ! ! ! ! 1 ! etc. ! UI
- 19. (1/2) ! SNS ! Twitter : 2.3 active user *1 ! Facebook : 11.9 active user *2 ! LINE : 2.0 user*3 ! ! Peer reviewed Journal 135 article / *4 ! 4 10% , 15 2 ! conference proceedings *1 http://weekly.ascii.jp/elem/000/000/084/84331/ *2 http://www.prnewswire.com/news-releases/facebook-reports-third- quarter-2013-results-229923821.html *3 http://en.lineblog.naver.jp/archives/30767259.html *4 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2909426/
- 23. tweet
- 25. ! ! ! ! ! ! !
- 26. 2.
- 28. ! !
- 31. 1/2 ! ! 1970 PC ! ! ! 2000 !
- 32. 2/2 ! ! RSS 2000 ! ! ! 2000 !
- 34. ! ! ! ! ! !
- 36. IoT
- 37. ! ! ! Google Now
- 38. ! ! <12:00 :25
- 39. ! ! ! ! ! ! !
- 40. ! ! ! ! ! ! !
- 41. 3.
- 43. ! ! 1993: [Brown+93] ! 1996: [Berger+96] ! 2001: [Lafferty+01] ! ! 2003: Latent Dirichlet Allocation [Blei+03] ! 2006: Pitman-Yor language model [Teh06] ! ! 2006: [Clarke+06][Riedel+06] ! 2010: [Koo+10][Rush+10] ! ! 2003: Neural language model [Bengio+03] ! 2010: Recurrent Neural Network [Mikolov+10] ! 2012: Skipgram Model (word2vec) [Mikolov+13]
- 45. ! ! ! !
- 46. 1. ! 2011: 30% # 20% ! 2012: 26% # 16% ! 2013: 11% # 2014: 6.7% http://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdf
- 47. 2. ! 2012/3: Google Hinton DNNresearch ! 2012/4: Baidu Institute of Deep Learning ! 2012/8, 10: Yahoo! IQ Engines LookFlow ! 2012/12: Facebook AI Lab LeCun ! 2014/1: Google DeepMind ! 2014/5: Andrew Ng Baidu ! 2014/8: IBM SyNAPSE
- 49. ! ! ! ! ! ! !
- 51. Neural Network Language Model (NNLM) [Bengio +03] ! N NN ! N-1
- 52. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] ! t-1 t ! NNLM N ! ! http://rnnlm.org
- 53. Skip-gram (word2vec) [Mikolov+13] ! ! CBOW ! Analogical reasoning ! word2vec
- 55. Dynamic Convolutional Neural Network [Kalchbrenner+14] ! CNN !
- 56. Long Short-Term Memory (LSTM) ! RNN ! ! tanh sigmoid sigmoid sigmoid
- 57. ! ! 1 !
- 58. ! libsvm, liblinear ! ! JUMAN, Chasen, MeCab ! ! Moses (GIZA++) ! ! Stanford CoreNLP ! ! word2vec ! Skipgram ! Theano, Caffe, cuda-convnet ! !
- 59. Caffe (http://caffe.berkeleyvision.org/) ! ! ! Convolutional Neural Network ! CPU GPU ! 1
- 60. ! ! 1 ! ! ! !
- 63. PFI 2010 Xappy 2012 ! 2012 Gunosy ! ! !
- 64. Xappy ! ! Blog ! Twitter Facebook SNS ! ! 4G ! ! ! etc. ! GUI
- 67. 40 vs 40
- 68. 2008-2011 ! ! : 40 ! : R&D ! : B2B ! : 2011- ! PFI ! : 40 ! : R&D ! : B2B ! :
- 70. ! ! ! ! ! ! !
- 71. ! ! ! IBM PFI
- 72. ! ! !
- 73. ! ! ! ! ! ! !
- 74. ! ! ! ! ! ! !
- 76. ! 90 !
- 78. ! ! !
- 81. 1 12
- 82. 503i 1 10KB iPhone6 1 10MB 12
- 83. CPU Pentium III 500MHz PC Apple A8 1.4GHz iPhone6 12
- 84. MD: 1 60 iPod shuffle: 500 30 12 wikipedia
- 85. ! ! ! !
- 87. ! ! !
- 88. ! I T π
- 89. 1 ! ! ! !
- 90. ! ! ! ! ! !
- 91. ! ! ! ! ! !
- 93. (1/4) ! [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics Vol. 19 (2), 1993. ! [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A. Della Pietra. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, Vol. 22 (1), 1996. ! [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. ICML2001.
- 94. (2/4) ! [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR Vol. 3, 2003. ! [Teh06] Yee Whye Teh. A Hierarchical Bayesian Language Model based on Pitman-Yor Processes. ACL 2006. ! [Clarke+06] James Clarke, Mirella Lapata. Constraint-Based Sentence Compression: An Integer Programming Approach. COLING/ACL 2006. ! [Riedel+06] Sebastian Riedel, James Clarke. Incremental Integer Linear Programming for Non-projective Dependency Parsing. COLING/ACL 2006.
- 95. (3/4) ! [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, David Sontag. Dual Decomposition for Parsing with Non-Projective Head Automata. EMNLP 2010. ! [Rush+10] Alexander M. Rush, David Sontag, Michael Collins, Tommi Jaakkola. On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing. EMNLP 2010. ! [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003.
- 96. (4/4) ! [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. ! [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013. ! [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix-Vector Spaces. EMNLP2012. ! [Kalchbrenner+14] Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom. A Convolutional Neural Network for Modelling Sentences. ACL2014.