![TOKENIZATION
• The raw text input is split into smaller pieces called tokens.
“I love coding” → [“I”, “love”, “coding”]
“unhappiness” → [“un”, “happiness”]
• Why is this important?
• Tokenization is crucial for preparing the text for numerical
processing. It helps the model to understand the boundaries
between different words or sub-words, which is essential
for capturing the semantics of the text.](https://image.slidesharecdn.com/llmpptpersonalfinal-241028152025-88482095/85/Non-technical-explanation-of-Large-Language-Model-6-320.jpg)
Non technical explanation of Large Language Model
- 1. LARGE LANGUAGE MODELS (LLMS) KEY CONCEPTS AND UNDERSTANDING
- 2. WHAT LLM IS NOT? • People often confuse LLMs with data storage due to the term 'large.' • LLMs are not databases. • LLMs are not search engines. • LLMs do not store facts like databases. • LLM is not copying and pasting.
- 3. WHAT IS LLM? • LLMs are deep learning models trained on large number of text data. • They generate human-like text based on language patterns. • LLMs contain billions of parameters and perform well on natural language tasks. • They learn to predict the next word in a sentence based on previous words
- 4. WHERE DOES LLM STAND? Artificial Intelligence (AI) is very a broad term, but generally it deals with intelligent machines. Machine Learning (ML) is a subfield of AI that specifically focuses on pattern recognition in data. Deep Learning is the field within ML that is focused on unstructured data, which includes text and images. Large Language Models (LLMs) deal with text specifically
- 6. TOKENIZATION • The raw text input is split into smaller pieces called tokens. “I love coding” → [“I”, “love”, “coding”] “unhappiness” → [“un”, “happiness”] • Why is this important? • Tokenization is crucial for preparing the text for numerical processing. It helps the model to understand the boundaries between different words or sub-words, which is essential for capturing the semantics of the text.
- 7. EMBEDDING • Each token is then converted into a numerical vector using a process called embedding. • Humans represent English words with a sequence of letters, like C-A-T for “cat.” Language models use a long list of numbers called a “word vector.” • Models are based on mathematical operations that require numerical inputs
- 8. CONTEXTUALIZATION • These vectors are processed through the neural network to understand the context. • For example, in the sentence “The cat sat on the mat,” the word “cat” is more closely related to “mat” than to “The.” • Contextualization allows the model to capture relationships between tokens.
- 9. OUTPUT GENERATION • The model uses a probability distribution to pick the most likely next token based on the context. • If the input is “How are you,” the model might generate an output like “I’m fine, thank you,” based on the probabilities of each token appearing after the input context. • Finally, the model generates an output text based on the processed context vectors
- 10. LLM MODELS AND THEIR SIZE
- 11. LLM PROMPTS An LLM prompt is a text input to a large language model (LLM) that instructs the model to generate a response.
- 14. CHAIN OF THOUGHT PROMPTING (COT)
- 15. COT WITH SHOT
- 17. ThankYou