Transformers - Part 1
1 like185 views
The document discusses the architecture and functioning of ChatGPT, specifically the generative pre-trained transformer model developed by OpenAI. It outlines the advantages of transformer models over recurrent neural networks (RNNs), focusing on concepts such as self-attention and the ethical implications associated with AI deployment. Topics covered include misuse, biases, and the impact of AI on human intelligence.
Transformers - Part 1
- 1. Transformers - Part 1 Dr. Akshika Wijesundara The science behind ChatGPT Research Associate King's College London | Consultant at National University of Singapore
- 2. Topics ● OpenAI and ChatGPT ● “Attention is all you need” ● RNNs vs Transformers ● Self-Attention ● Transformer architecture ● Ethical implications and challenges
- 3. OpenAI is an AI research and deployment company with the mission to ensure that artificial general intelligence benefits all of humanity.
- 6. How is ChatGPT built? ● GPT (Generative Pre-trained Transformer) - 3 is an autoregressive language model (175B parameters). ● Later InstructGPT was developed to address the shortcomings of GPT3. ● InstructGPT uses human feedback to reinforce the model (1.3B parameters)
- 8. Transformers > RNN* RNN Transformers Process a sequence of data in parallel 😕 😃 Handle long-term dependencies 😐 😃 Capturing global dependencies 😐 😃 Self-attention 😕 😃
- 10. Attention Sam wants to visit Greece but she is poor
- 11. Attention Sam wants to visit Greece but she is poor
- 12. Attention
- 21. Ethical & Legal Implications
- 22. Ethical & Legal Implications ● Misuse ○ Fake news and bots that spread it ○ Using the tool in exams ● Biases ● Impact on human intelligence ● Copyright infringement
- 23. What we covered ● Overview of ChatGPT ● Transformers ● Self-Attention ● Ethical implications of ChatGPT