Natural Language Processing (Almost) from Scratch（第 6 回 Deep Learning 勉強会資料; 榊）
Download as PPTX, PDF10 likes5,659 views
Deep Learning Japan @ 東大です http://www.facebook.com/DeepLearning https://sites.google.com/site/deeplearning2013/
Natural Language Processing (Almost) from Scratch（第 6 回 Deep Learning 勉強会資料; 榊）
- 1. Natural Language Processing (Almost) from Scratch Ronan Collobert et al. Journal of Machine Learning Research vol.12 (2011)
- 2. 本論文の選定理由 • ACL 2012 Tutorial Deep Learning for NLPにて紹介さ れている • 代表的なNLPタスクにDeep Learningを適用している – POS tagging – Chunking – Named Entity Recognition – Semantic Role Labeling • NLP with Deep Learningの代表的な研究者が執筆し ている – Chris Manning – Ronan Collobert
- 3. 本論文のまとめ 目的 Propose a unified neural network architecture and learning algorithm that can be applied to various NLP tasks POS tagging, Chunking, NER, SLR 結論 人手でfeatureを作成する代わりに、大量のlabeled/unlabeled training dataからinternal representationを学習する 本研究の成果は、高精度で低計算コストなfreely available tagging systemを構築するための基礎となる
- 4. 本論文のまとめ 注目点 様々なNLPタスクにNeural Networkを適用する際 に、どのようにデータを扱うべきか Labeled Data/Unlabeled Dataにおける扱いの違い について
- 6. 背景と目的 目的 task-specific engineeringせずに、複数の基準手法 を超えることを目指す large unlabeled data setsから発見される intermediate representationを適用することで、 多くのNLPタスクについて高精度を得ることを 目指す Multi-tasking な言語モデルを構築する
- 7. Multi Tasking: shared features
- 8. タスクとデータセット
- 9. タスク説明 • Part Of Speech tagging – 各単語、形態素への品詞付与 • Chunking – 名詞句、動詞句、専門用語等文法的にひとま とまりとして扱われるword sequence の抽出 • Named Entity Recognition – 固有名詞抽出（地名、人名など）
- 10. タスク説明 • Semantic Role Labeling – 文法的役割（主語、目的語、述語）や語同士 の係り受け関係など、意味的や役割を付与す る
- 12. Chapter 3 The Networks
- 13. 提案手法 問題設定 全てのNLPタスクは語へのラベル付けであると 考える Traditional Approach hand-designed featuresを分類アルゴリズムに適用 New Approach multilayer neural networkによる学習
- 14. 提案手法 • Transforming word into Feature Vectors • Extracting Higher Level Features from Word Feature Vectors • Training • Benchmark Result
- 15. 提案手法 • Transforming word into Feature Vectors • Extracting Higher Level Features from Word Feature Vectors • Training • Benchmark Result
- 16. Neural Networks
- 17. 提案手法〜概要〜 Window approach network Sentence approach network
- 19. 提案手法 • Transforming word into Feature Vectors • Extracting Higher Level Features from Word Feature Vectors • Training • Benchmark Result
- 20. Extracting Higher Level Features From Word Feature Vectors L層のNeural Network l層関数 パラメータ
- 22. Linear Layer Window approach Parameters to be trained 第l層でのhidden unit数
- 23. HardTanh Layer • Non-linear featureの表現 Window approach
- 26. Time Delay Neural Network
- 28. Max Layer Sentence approach 各hidden unit ごとにt=0〜tで最大となる重みを第l層 への重みに
- 29. Tagging Schemes
- 30. 提案手法 • Transforming word into Feature Vectors • Extracting Higher Level Features from Word Feature Vectors • Training • Benchmark Result
- 32. Training Word Level Log-Likelihood soft max all over tags
- 33. Training Sentence Level Log-Likelihood transition score to jump from tag k to tagi Sentence score for a tag path
- 34. Training Sentence Level Log-Likelihood Conditional likelihood by normalizingw.r.tall possible paths
- 35. Training 正規化項はrecursive Forward algorithm で算出可能 Inference: Viterbi algorithm (replace logAdd by max)
- 36. 提案手法 • Transforming word into Feature Vectors • Extracting Higher Level Features from Word Feature Vectors • Training • Benchmark Result
- 37. Pre Processing • use lower case words in the dictionary • add “caps” feature to words had at least one non-initial capital letter • number with in a word are replace with the string “NUMBER”
- 38. Hyper-parameters
- 39. Benchmark Result Sentences with similar words should be tagged in the same way. The cat sat on the mat The feline sat on the mat
- 41. Chapter 4 Lots of Unlabeled Data
- 43. Lots of Unlabeled Data • Two window approach (11) networks (100HU) trained on two corpus • LM1 – Wikipedia: 631 Mwords – order dictionary words by frequency – increase dictionary size: 5000, 10; 000, 30; 000, 50; 000, 100; 000 – 4 weeks of training • LM2 – Wikipedia + Reuter=631+221=852M words – initialized with LM1, dictionary size is 130; 000 – 30,000 additional most frequent Reuters words – 3 additional weeks of training
- 46. Chapter 5 Multitask Learning
- 48. Multitask Learning window approachでは、First Layerのパラメータを共 有 Joint Training
- 51. その他の工夫 • Suffix Features – Use last two characters as feature • Gazetters – 8,000 locations, person names, organizations and misc entries from CoNLL2003 • POS – use POS as a feature for CHUNK &NER • CHUNK – use CHUNK as a feature for SRL
- 52. その他の工夫
- 54. Conclusion • Achievements – “All purpose" neural network architecture for NLP tagging – Limit task-specic engineering – Rely on very large unlabeled datasets – We do not plan to stop here • Critics – Why forgetting NLP expertise for neural network training skills? • NLP goals are not limited to existing NLP task • Excessive task-specic engineering is not desirable – Why neural networks? • Scale on massive datasets • Discover hidden representations • Most of neural network technology existed in 1997 (Bottou, 1997)