From the course: Introduction to Transformer Models for NLP
Join today to access over 24,800 courses taught by industry experts.
GPT for style completion
From the course: Introduction to Transformer Models for NLP
GPT for style completion
- Section 8.1: GPT for style completion. We've talked in the past how pre-training GPT, 2 at least, consisted of running the language modeling task across a very, very large corpus of text called WebText, created from getting outbound links from Reddit with more than three karma, and then performing the auto-regressive language model to teach GPT how people speak on the internet. But now it's time to turn our attention to fine-tuning GPT on a brand new corpus. This is a very common technique for teaching GPT how words are used in different domains. So for our example today, we're going to be taking one of my textbooks and fine-tuning GPT to read it a few times and understand how those words are used in the context of data science. So let's jump right into our code. To fine-tune GPT-2 for style completion on a brand new corpus, we'll have a few imports. Many of these should look familiar by now, but let's still go through them. From our handy dandy transformers library, we're going to…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.