From the course: LLaMa for Developers
Join today to access over 24,700 courses taught by industry experts.
Fine-tuning with LLaMA using LoRa - Llama Tutorial
From the course: LLaMa for Developers
Fine-tuning with LLaMA using LoRa
- [Instructor] In the past video, we learned how to fine-tune a LLaMA model by freezing some layers. In this video, we're going to learn a technique called LoRa. LoRa has been quite popular recently, so let's go through the original paper and see how it works. Now the LoRa paper was introduced in 2021, focused on the ability to train larger models like GPT-3. The benefits were that you can fine-tune the model by adding fewer new trainable parameters, having higher training throughput, and having no additional inference latency. Now, quick architecture on how LoRA works. LoRA works by connecting to the linear layers of a large language model. These are fully-connected neural networks. Then, by projecting the information into our lower-ranked space, it's able to learn this information and combine the output at the end. Now, this paper is more technical than that, but that's the key idea. Now, if we keep scrolling, the…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.