Transformer and BERT
Download as PPTX, PDF0 likes785 views
This document discusses various methods for modeling long sequences, including LSTMs, CNNs, attention mechanisms, and Transformers. It provides details on the Transformer architecture such as multi-head attention, feed-forward layers, and computational complexity. BERT is introduced as a Transformer-based model used for natural language understanding tasks. Details are given on BERT's training methodology and the resources required to train BERT-base.
![BERT (training tasks)
Masked Language Model: masked word with the [MASK] token
Next Sentence Prediction](https://image.slidesharecdn.com/ibtd1zwnqggum1eu86h0-signature-75525836ffdd7653e07cc98c4424e2a1474784c088484fc6c26d5e97181bd851-poli-200806110847/85/Transformer-and-BERT-16-320.jpg)
Transformer and BERT
- 1. Transformer & Bert Models for long sequence
- 2. How to model long sequence (LSTM) From: https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714
- 3. How to model long sequence (CNN) From: http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/
- 4. How to model long sequence (CNN) Convolutional Sequence to Sequence Learning Neural Machine Translation of Rare Words with Subword Units Google's Neural Machine Translation System
- 6. Attention Mechanism NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE
- 7. Transformer (Q, K, V) From: http://jalammar.github.io/illustrated-transformer/
- 9. What about order ? From: http://jalammar.github.io/illustrated-transformer/ From: https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
- 11. Transformer (parameters) Multi-Head-Attention: (512 * 64 * 3 * 8) + (8 * 64 * 512) Feed-Forward: (512*2048) + 2048 + (2048 * 512) + 512 Last-Linear-Layer: (512 * 370000) Total: Multi-Head-Attention * 3 * 6 + Feed-Forward * 2 * 6 + Last-Linear-Layer = 63 * 1e6 ((512*64*3*8)+(8*64*512)) * 18 + ((512*2048)+(2048*512)+2048+512) * 12 + 512 * 37000
- 12. Transformer (FLOPS per token) Multi-Head-Attention: ((512+511)*64)*3*8+((512+511)*512) Feed-Forward: ((512+511)*2048)+2048+((2048+2047)*512)+512 Last-Linear-Layer: ((512+511)*370000)+370000 Total: Multi-Head-Attention * 3 * 6 + Feed-Forward * 2 * 6 + Last-Linear-Layer = 467MFLOPS (((512+511)*64)*3*8+((512+511)*512))*18+(((512+511)*2048)+2048+((2048+2047)*512)+512)*12+((512+511)*370000)+370000
- 16. BERT (training tasks) Masked Language Model: masked word with the [MASK] token Next Sentence Prediction
- 17. BERT BERT-base: L=12, H=768, A=12, Total Parameters: 110M Batch-size: 256 sequences (256 sequences * 512 tokens = 128000 tokens/batch), for 1M steps. 128000 * 467M FLOPS = 60 TFLOPS Training BER-base on 4 TPUs pod (16 TPU chips total), took 4 days to complete Conclusion Space: 440MB + 393MB = 833MB Speed: 173 TFLOPS per second
- 18. From Paper: Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction
- 19. Some thoughts All matrix add/multiple operations (a slight bit of sin/cos/exp) More hardware-friendly Model Big Op (automatically) Transformer + NTM