![CloudKarya
Adding Special Context Tokens
● Need for special tokens
○ To handle unknown words <|unk|>
○ To identify start and end of the text
○ To pad the shorter texts to match the length of longer texts
● Popular tokens used
○ [BOS] (beginning of sequence)
○ [EOS](end of sequence)
○ [PAD](padding)
● The tokenizer for GPT models uses only <|endoftext|>
● GPT models handle unknown words using BPE](https://image.slidesharecdn.com/buildalargelanguagemodel-250305140212-e59bb8bd/85/Building-a-large-language-models-from-scratch-pdf-13-320.jpg)
Building a large language models from scratch .pdf
- 1. Build a Large Language Model Understanding LLMs
- 2. CloudKarya 1.1 What is an LLM? ● Neural network that can understand,generate and respond like human ● Trained on large data ● “Large” in “Large language Model” refers to a) Model size(parameters) b)dataset ● Utilizes transformer architecture with attention mechanism ● Also known as Gen AI because of their generative capabilities Artificial Intelligence Machine Learning Deep Learning Large Language Models Gen AI
- 3. CloudKarya 1.2 Applications of LLMs ● Machine Translation ● Text Summarization ● Sentiment Analysis ● Content Creation ● Code generation ● Conversational agents like chatbots
- 4. CloudKarya 1.3 Stages of building and using LLMs ● Data preparation ● Pretraining LLM on large unlabelled text data ○ This will have text completion and few shot capabilities ● Preparing the labelled datasets for specific tasks ● Train LLM’s on these task specific datasets to get a fine-tuned LLM ○ Classification ○ Summarization ○ Translation ○ Personal assistant ● Finetuning has 2 types ○ Instruction fine-tuning ○ Classification fine-tuning
- 5. CloudKarya 1.4 Introducing the Transformer Architecture ● Original Transformer ○ Developed for machine translation , English to German ● Encoder ○ Processes the input text and produces an embedding representation ● Decoder ○ The output from encoder can be used by decoder to generate the translated text one word at time ● Self-attention mechanism ● BERT (encoder based model) ○ Masked language modeling ○ X uses Bert ● GPT (Decoder only model) ○ Auto regressive model
- 6. CloudKarya 1.5 Utilizing large datasets ● Huge corpus with billions of words ● Common datasets ○ CommonCrawl ○ WebText2 ○ Books1 ○ Book2 ○ Wikipedia ● GPT training datasets were not released ● Dolma
- 7. CloudKarya 1.6 A closer look at the GPT architecture ● Decoder -only architecture ● Auto regressive Models ● GPT-3 has 96 transformer layers and 175 billion parameters ● Emergent behavior
- 8. CloudKarya 1.7 Building a LLM ● Stage 1 ○ Building an LLM ■ Data Preparation and Sampling ■ Attention mechanism ■ LLM architecture ● Stage 2 ○ Foundational model ■ Training loop ■ Model evaluation ■ Load pretrained weights ● Stage 3 ○ Fine tuning ■ Classifier ■ Personal assistant
- 9. Build a Large Language Model Working With Text Data
- 10. CloudKarya Understanding Word Embeddings ● Embedding: Converting data into a vector format. ● Types of embeddings ○ Text, Audio, Video ● Types of text embeddings ○ Word, Sentence, Paragraphs (RAG) ○ Whole documents ● Word2Vec ○ Similar context - same embedding ● Models for word embeddings ○ Static Models (Word2Vec, GloVe, FastText) ○ Contextual Models ( BERT, GPT, etc) ● LLMs produce their own embeddings which are updated during training. ● GPT-2 - 768 dimensions, GPT-3- 12,288 dimensions Discrete Objects Continuous Space Nonnumeric Machine Readable
- 11. CloudKarya Tokenizing Text ● 1st step in creating embeddings ● Tokens ○ Individual words or special characters, including punctuations. ● LLM Training ○ The-verdict - a short story by Edith Wharton ○ Goal - tokenize 20,479 Character short story I love reading books. I love reading books . Input Text Tokenized Text
- 12. CloudKarya Converting Tokens into Token ID’s ● Intermediate step before converting tokens into embeddings ● Vocabulary ○ Defines how we map each unique word and special character to a unique integer ○ The vocab size for The-verdict is 1,130
- 13. CloudKarya Adding Special Context Tokens ● Need for special tokens ○ To handle unknown words <|unk|> ○ To identify start and end of the text ○ To pad the shorter texts to match the length of longer texts ● Popular tokens used ○ [BOS] (beginning of sequence) ○ [EOS](end of sequence) ○ [PAD](padding) ● The tokenizer for GPT models uses only <|endoftext|> ● GPT models handle unknown words using BPE
- 14. CloudKarya Byte Pair Encoding ● It is famous tokenization technique used to train GPT-2, GPT-3, RoBERTa, BART, and DeBERTa. ● Training phase ○ BPE learns a vocabulary of subwords by iteratively merging the most frequent character pairs. ● Tokenization Phase ○ Split text into characters. ○ Iteratively match the longest possible subwords from the vocabulary. ○ Replace matched subwords with their corresponding token IDs. ● Tiktoken ○ An open source python library used for implementing BPE. ○ BPE tokenizer used for GPT-2 and GPT-3 have a vocabulary size of 50,257 ● Handling unknown words ○ Unknown words are breakdown into individual characters ensures that LLM can process any text.
- 15. CloudKarya Data Sampling With a Sliding Window ● LLMs prediction task is to predict the next word that follows the input block ● Input - target pairs needs to be created ● To perform data sampling ○ We make use of PyTorch’s built-in Dataset and DataLoader classes. ○ Hyperparameters for DataLoader ■ Batch_size, max_legth, stride, num_workers
- 16. CloudKarya Creating Token Embeddings ● Last step in preparing input text for LLM training ● Token ids are converted to embeddings ○ These embeddings are initialized with random values ○ This serves as a starting point for LLMs learning process ● Using torch.nn.Embedding create a embedding layer ○ This embedding layer is a lookup operation that retrieves rows from the embedding layer’s weight matrix ●
- 17. CloudKarya Encoding Word Positions ● Need ○ Self-attention mechanism doesn’t have a notion of position order for the tokens within a sequence ○ Embedding layer will return same embedding for same token ID every time irrespective of the position. ● So we inject the positional encoding to add positional information ● It is of two types ○ Absolute Positional Embeddings ○ Relative Positional Embeddings ● OpenAI’s GPT models use absolute positional embeddings ● These embeddings are optimized during the training process ● The dimensions of positional encoding will be batch_size x context_length