From the course: Introduction to Transformer Models for NLP
Join today to access over 24,800 courses taught by industry experts.
How language models look at text
From the course: Introduction to Transformer Models for NLP
How language models look at text
- Section 1.4, "How Language Models Look At Text." Back in the beginning, I mentioned how in 2001, a language modeling task was solved for the first time using a deep learning feedforward architecture, but I didn't really talk about what a language modeling task is. Now, it's actually important we take a look at that because language models and language modeling tasks are the core tenant of how transformers and transformer-based architectures learn language, and more specifically, language rules, how words are used in sentences, how sentences are treated in a larger corpora. A language modeling task, a model is trained to predict a missing word, or token, in a sequence of words, or tokens. Now, in general, there's two kinds of language models, auto-regressive and auto-encoding. Consider the following example, "If you don't," blank, "at the sign, you will get a ticket." Now, if you're watching this at home, you're probably already filling in the blank yourself, but this is actually a…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.