SlideShare a Scribd company logo
Effective Approaches to Attention-based
Neural Machine Translation
Minh-Thang Luong Hieu Pham Christopher D. Manning
Stanford University
Sep 2015
2
Attention
(NMT: neural machine translation)
2 (groval local)
local a9en:on
⚔
|
3
encoder decoder
→sec2sec
encoder-decoder
RNN
| Attention
4
5
Attention
⚔ A'en(on
6
A"en%on
Global approach Local approach
A"en%on Grobal a"en%on Local a"en%on
Global attention
7
Decoder
Attention t
ht
at
ct yt
Global attention
8
Local a'en*on
9
Local attention
10
grobal attention
11
WMT ⇄
BLEU[1]
BLEU:
bilingual evaluation understudy
NMTの精度の評価手法のひとつ
言葉の境界を持たない言語を扱うことはできない
Traningデータ:
WMT 14 trainingdata
consisting of 4.5M sentences pairs
English words …116M
German words…110M
WMT:
Ninth Workshop on Statistical
Machine Translation
機械翻訳のワークショップ
ACL主催??
12
https://github.com/lmthang/nmt.hybrid
13
Attention
① a)en*on
② local attention
- WMT 5.0BLUE
- WMT’14 WMT’15
① | WMT BLUE
14
② |
15
Conclusion |
16
-
Attention
-
①local a1en2on
②WMT’14,WMT’15
-
a1en2on
-
WMT BLEU

More Related Content

PPTX
NLPにおけるAttention~Seq2Seq から BERTまで~
PPTX
[DL輪読会]BERT: Pre-training of Deep Bidirectional Transformers for Language Und...
PDF
SSII2022 [TS1] Transformerの最前線〜 畳込みニューラルネットワークの先へ 〜
PPTX
[DL輪読会]ドメイン転移と不変表現に関するサーベイ
PPTX
有向グラフに対する 非線形ラプラシアンと ネットワーク解析
PPTX
モデル高速化百選
PDF
最近のDeep Learning (NLP) 界隈におけるAttention事情
PDF
BERT+XLNet+RoBERTa
NLPにおけるAttention~Seq2Seq から BERTまで~
[DL輪読会]BERT: Pre-training of Deep Bidirectional Transformers for Language Und...
SSII2022 [TS1] Transformerの最前線〜 畳込みニューラルネットワークの先へ 〜
[DL輪読会]ドメイン転移と不変表現に関するサーベイ
有向グラフに対する 非線形ラプラシアンと ネットワーク解析
モデル高速化百選
最近のDeep Learning (NLP) 界隈におけるAttention事情
BERT+XLNet+RoBERTa

What's hot (20)

PDF
【論文紹介】Seq2Seq (NIPS 2014)
PPTX
[DL輪読会]MetaFormer is Actually What You Need for Vision
PPTX
【DL輪読会】The Forward-Forward Algorithm: Some Preliminary
PPTX
[DL輪読会]Life-Long Disentangled Representation Learning with Cross-Domain Laten...
PDF
ConvNetの歴史とResNet亜種、ベストプラクティス
PPTX
近年のHierarchical Vision Transformer
PPTX
[DL輪読会]Neural Ordinary Differential Equations
PDF
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
PDF
論文紹介「PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet」
PDF
全力解説!Transformer
PDF
BERTに関して
PDF
【DL輪読会】Hierarchical Text-Conditional Image Generation with CLIP Latents
PDF
【DL輪読会】How Much Can CLIP Benefit Vision-and-Language Tasks?
PDF
[DL輪読会]ICLR2020の分布外検知速報
PPTX
【DL輪読会】Contrastive Learning as Goal-Conditioned Reinforcement Learning
PDF
20190706cvpr2019_3d_shape_representation
PDF
方策勾配型強化学習の基礎と応用
PPTX
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
PDF
CycleGANによる異種モダリティ画像生成を用いた股関節MRIの筋骨格セグメンテーション
PDF
SSII2021 [SS1] Transformer x Computer Visionの 実活用可能性と展望 〜 TransformerのCompute...
【論文紹介】Seq2Seq (NIPS 2014)
[DL輪読会]MetaFormer is Actually What You Need for Vision
【DL輪読会】The Forward-Forward Algorithm: Some Preliminary
[DL輪読会]Life-Long Disentangled Representation Learning with Cross-Domain Laten...
ConvNetの歴史とResNet亜種、ベストプラクティス
近年のHierarchical Vision Transformer
[DL輪読会]Neural Ordinary Differential Equations
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
論文紹介「PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet」
全力解説!Transformer
BERTに関して
【DL輪読会】Hierarchical Text-Conditional Image Generation with CLIP Latents
【DL輪読会】How Much Can CLIP Benefit Vision-and-Language Tasks?
[DL輪読会]ICLR2020の分布外検知速報
【DL輪読会】Contrastive Learning as Goal-Conditioned Reinforcement Learning
20190706cvpr2019_3d_shape_representation
方策勾配型強化学習の基礎と応用
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
CycleGANによる異種モダリティ画像生成を用いた股関節MRIの筋骨格セグメンテーション
SSII2021 [SS1] Transformer x Computer Visionの 実活用可能性と展望 〜 TransformerのCompute...
Ad

Similar to [論文読み]Effective Approaches to Attention-based Neural Machine Translation (11)

PPTX
Notes on attention mechanism
PDF
Topic 4: The Magician's Hat: Turning Data into Business Intelligence (3)
PPTX
Natural Language Processing - Research and Application Trends
PPTX
Machine Tanslation
PDF
Deep Learning for Machine Translation - A dramatic turn of paradigm
PDF
NLP using transformers
PPTX
Introduction to Transformer Model
PPTX
Mise14 @ ICSE1 14 Uncertainty in Bidirectional Transformations
PPTX
Neural Machine Translation in the NLP.pptx
PDF
Advanced Neural Machine Translation (D4L2 Deep Learning for Speech and Langua...
PDF
Big Data Spain 2017 - Deriving Actionable Insights from High Volume Media St...
Notes on attention mechanism
Topic 4: The Magician's Hat: Turning Data into Business Intelligence (3)
Natural Language Processing - Research and Application Trends
Machine Tanslation
Deep Learning for Machine Translation - A dramatic turn of paradigm
NLP using transformers
Introduction to Transformer Model
Mise14 @ ICSE1 14 Uncertainty in Bidirectional Transformations
Neural Machine Translation in the NLP.pptx
Advanced Neural Machine Translation (D4L2 Deep Learning for Speech and Langua...
Big Data Spain 2017 - Deriving Actionable Insights from High Volume Media St...
Ad

Recently uploaded (20)

PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
PDF
Heart disease approach using modified random forest and particle swarm optimi...
PPTX
Programs and apps: productivity, graphics, security and other tools
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
Encapsulation theory and applications.pdf
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
NewMind AI Weekly Chronicles - August'25-Week II
PDF
Machine learning based COVID-19 study performance prediction
PPTX
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PPTX
1. Introduction to Computer Programming.pptx
PDF
Empathic Computing: Creating Shared Understanding
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
PDF
Accuracy of neural networks in brain wave diagnosis of schizophrenia
PPTX
Machine Learning_overview_presentation.pptx
PDF
August Patch Tuesday
PDF
Assigned Numbers - 2025 - Bluetooth® Document
PDF
Encapsulation_ Review paper, used for researhc scholars
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
Heart disease approach using modified random forest and particle swarm optimi...
Programs and apps: productivity, graphics, security and other tools
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Encapsulation theory and applications.pdf
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Reach Out and Touch Someone: Haptics and Empathic Computing
NewMind AI Weekly Chronicles - August'25-Week II
Machine learning based COVID-19 study performance prediction
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
1. Introduction to Computer Programming.pptx
Empathic Computing: Creating Shared Understanding
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Group 1 Presentation -Planning and Decision Making .pptx
Accuracy of neural networks in brain wave diagnosis of schizophrenia
Machine Learning_overview_presentation.pptx
August Patch Tuesday
Assigned Numbers - 2025 - Bluetooth® Document
Encapsulation_ Review paper, used for researhc scholars

[論文読み]Effective Approaches to Attention-based Neural Machine Translation