Attention-based Models (DLAI D8L 2017 UPC Deep Learning for Artificial Intelligence)
3 likes1,217 views
The document discusses advancements in attention models, particularly their simplicity and effectiveness in deep learning applications such as machine translation and natural language processing. It covers the mechanisms of attention, variations like hard and monotonic attention, and improvements to enhance model performance. The presentation highlights the versatility of attention models across various tasks, including chatbots, image captioning, and speech recognition.
![Outline
1. Sequence modeling & Sequence-to-sequence models [WRAP-UP FROM PREVIOUS RNN’s SESSION]
2. Attention-based mechanism
3. Attention varieties
4. Attention Improvements
5. Applications
6. “Attention is all you need”
7. Summary
3](https://image.slidesharecdn.com/dlai2017d9lattention-171121141428/85/Attention-based-Models-DLAI-D8L-2017-UPC-Deep-Learning-for-Artificial-Intelligence-3-320.jpg)
![Quizz
1.Mark all statements that are true
A. Sequence modeling only refers to language applications
B. The attention mechanism can be applied to an encoder-decoder architecture
C. Neural machine translation systems require recurrent neural networks
D. If we want to have a fixed representation (thought vector), we can not apply attention-based
mechanisms
2. Given the query vector q=[], the key vector 1 k1=[] and the key vector 2 k2=[].
A. What are the attention weights 1 & 2 computing the dot product?
B. And when computing the scaled dot product?
C. To what key vector are we giving more attention?
D. What is the advantage of computing the scaled dot product?
52](https://image.slidesharecdn.com/dlai2017d9lattention-171121141428/85/Attention-based-Models-DLAI-D8L-2017-UPC-Deep-Learning-for-Artificial-Intelligence-52-320.jpg)
Attention-based Models (DLAI D8L 2017 UPC Deep Learning for Artificial Intelligence)
- 4. Sequence modeling Model the probability of sequences of words From previous lecture… we model sequences ith RNNs p(I’m) p(fine|I’m) p(.|fine) EOS I’m fine .<s> 4
- 5. Sequence-to-sequence models how are you ? Cómo estás EOS encoder decoder ¿ Cómo estás ? ? ¿ <s> THOUGHT/ CONTEXT VECTOR 5
- 7. 7
- 11. Attention encoder decoder + Attention allows to use multiple vectors, based on the length of the input 11
- 13. Attention computation I • Use “query” vector (decoder state) and “key” vectors (all encoder states) • For each query-key pair, calculate weight • Normalize to add to one using softmax Query Vector Key Vectors a1=2.1 a2=-0.1 a3=0.3 a4=-1.0 softmax a1=0.5 a2=0.3 a3=0.1 a4=0.1 13
- 14. Attention computation II • Combine together value vectors (usually encoder states, like key vectors) by taking the weighted sum Value Vectors a1=0.5 a2=0.3 a3=0.1 a4=0.1 * * * * 14
- 15. Attention Score Functions q is the query and k is the key Reference Multi-layer Perceptron 𝑎 𝑞, 𝑘 = tanh ( 𝒲- 𝑞, 𝑘 ) Flexible, often very good with large data Bahdanau et al., 2015 Bilinear 𝑎 𝑞, 𝑘 = 𝑞/ 𝒲𝑘 Luong et al 2015 Dot Product 𝑎 𝑞, 𝑘 = 𝑞/ 𝑘 No parameters! But requires sizes to be the same Luong et al. 2015 Scaled Dot Product 𝑎 𝑞, 𝑘 = 𝑞/ 𝑘 |𝑘| Scale by size of the vector Vaswani et al. 2017 15
- 23. Multi-headed Attention I Multiple attention “heads” focus on different parts of the sentence 𝑎 𝑞, 𝑘 = 𝑞/ 𝑘 |𝑘| 23
- 24. Multi-headed Attention II Multiple attention “heads” focus on different parts of the sentence E.g. Multiple independently learned heads (Vaswani et al. 2017) 𝑎 𝑞, 𝑘 = 𝑞/ 𝑘 |𝑘| 24
- 26. Coverage Problem: Neural models tends to drop or repeat content In MT, 1.Over-translation: some words are unnecessarily translated for multiple times; 2. Under-translation: some words are mistakenly untranslated. SRC: Señor Presidente, abre la sesión. TRG: Mr President Mr President Mr President. Solution: Model how many times words have been covered e.g. maintaining a coverage vector to keep track of the attention history (Tu et al., 2016) 26
- 31. Chatbots a computer program that conducts a conversation Human: what is your job Enc-dec: i’m a lawyer Human: what do you do ? Enc-dec: i’m a doctor . what is your job I’m a EOS <s> I’m a lawyer lawyer + attention 31
- 33. Other NLP Tasks Text summarization: process of shortening a text document with software to create a summary with the major points of the original document. Question Answering: automatically producing an answer to a question given a corresponding document. Semantic Parsing: mapping natural language into a logical form that can be executed on a knowledge base and return an answer Syntactic Parsing: process of analysing a string of symbols, either in natural language or in computer languages, conforming to the rules of a formal grammar 33
- 34. Image captioning I decoder encoder A cat on the mat a cat <s> a on the mat cat on the 34
- 36. Other Computer Vision Tasks with Attention Visual Question Answering: given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Video Caption Generation: attempts to generate a complete and natural sentence, enriching the single label as in video classification, to capture the most informative dynamics in videos. 36
- 46. 7. Summary 46
- 49. Attention-based mechanisms Soft vs Hard: soft attention weights all pixels, hard attention crops the image and forces attention only on the kept part. Global vs Local: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. Intra vs External: intra attention is within the encoder’s input sentence, external attention is across sentences. 49
- 50. One large encoder-decoder •Text, speech, image… is all converging to a signal paradigm? •If you know how to build a neural MT system, you may easily learn how to build a speech-to-text recognition system... •Or you may train them together to achieve zero-shot AI. *And other references on this research direction…. 50
- 51. 51 Research going on… Interested? marta.ruiz@upc.edu Q&A ?