SlideShare a Scribd company logo
1 | © Copyright 2024 Zilliz
1
Yujian Tang | Zilliz
Introduction to LLM
Customization
2 | © Copyright 2024 Zilliz
2
Yujian Tang
Senior Developer Advocate, Zilliz
yujian@zilliz.com
https://www.linkedin.com/in/yujiantang
https://www.twitter.com/yujian_tang
Speaker
3 | © Copyright 2024 Zilliz
3
01 Introduction to LLMs
CONTENTS
03
04 Examples
02 RAG
Fine Tuning
4 | © Copyright 2024 Zilliz
4
01 Introduction to LLMs
5 | © Copyright 2024 Zilliz
5
A Basic Neural Net
6 | © Copyright 2024 Zilliz
6
A Recurrent Neural Network
7 | © Copyright 2024 Zilliz
7
A Transformer Architecture
8 | © Copyright 2024 Zilliz
8
GPT
9 | © Copyright 2024 Zilliz
9
02 RAG
10 | © Copyright 2024 Zilliz
10
RAG
RAG
Inject your data via a vector
database like Milvus/Zilliz
Query LLM
Milvus
Your Data
Primary Use Case
- Factual Recall
- Forced Data Injection
- Cost Optimization
Embed
11 | © Copyright 2024 Zilliz
11
What Does Vector Data Look Like?
12 | © Copyright 2024 Zilliz
12
Find Semantically Similar Data
Apple made profits of $97 Billion in 2023
I like to eat apple pie for profit in 2023
Apple’s bottom line increased by record numbers in 2023
13 | © Copyright 2024 Zilliz
13
But wait! There’s more!
14 | © Copyright 2024 Zilliz
14
RAG lets us inject data via semantic similarity
provided by vector databases like Milvus
15 | © Copyright 2024 Zilliz
15
03 Fine Tuning
16 | © Copyright 2024 Zilliz
16
RAG vs Fine Tuning
LLM
Fine Tuning
Augment an LLM by training it
on your data
Your Data
“New” LLM
Query
Primary Use Case
- Style transfer
- Domain specific usage
17 | © Copyright 2024 Zilliz
17
18 | © Copyright 2024 Zilliz
18
19 | © Copyright 2024 Zilliz
19
Types of Fine Tuning
- Full fine tuning
- LoRA
- QLoRA
20 | © Copyright 2024 Zilliz
20
Fine Tuning Methods
- Supervised Fine Tuning
- Direct Preference Optimization
- 𝚿 (Identity) Preference Optimization
- Odds-Ratio Preference Optimization
21 | © Copyright 2024 Zilliz
21
Fine tuning lets us train LLMs to operate in certain
styles or domains
22 | © Copyright 2024 Zilliz
22
04 Examples
Give Milvus a
Star!
23 | © Copyright 2024 Zilliz
23
RAG without OpenAI project
24 | © Copyright 2024 Zilliz
24
AI Agents Projects
25 | © Copyright 2024 Zilliz
25
Fine Tuning Library
26 | © Copyright 2024 Zilliz
26
Start building
with Zilliz Cloud today!
zilliz.com/cloud

More Related Content

PDF
2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Te...
PDF
17-October-2024 NYC AI Camp - Step-by-Step RAG 101
PDF
11-OCT-2024_AI_101_CryptoOracle_UnstructuredData
PDF
2025-02-24 - AWS meetup - Zilliz presentation.pdf
PDF
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
PDF
10-25-2024_BITS_NYC_Unstructured Data and LLM_ What, Why and How
PDF
NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)
PDF
08-13-2024 NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)
2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Te...
17-October-2024 NYC AI Camp - Step-by-Step RAG 101
11-OCT-2024_AI_101_CryptoOracle_UnstructuredData
2025-02-24 - AWS meetup - Zilliz presentation.pdf
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
10-25-2024_BITS_NYC_Unstructured Data and LLM_ What, Why and How
NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)
08-13-2024 NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)

Similar to Introduction to Large Language Model Customization.pdf (20)

PDF
Supercharge Spark: Unleashing Big Data Potential with Milvus for RAG systems
PDF
2025-04-05 - Block71 Event - The Landscape of GenAI and Ecosystem.pdf
PDF
Zilliz Cloud Demo for performance and scale
PDF
09-19-2024 AI Camp Hybrid Seach - Milvus for Vector Database
PDF
Scaling Vector Search: How Milvus Handles Billions+
PDF
Building Production Ready Search Pipelines with Spark and Milvus
PDF
Multimodal Retrieval-Augmented Generation (RAG) with Vector Database
PDF
Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI...
PDF
09-25-2024 NJX Venture Summit Introduction to Unstructured Data
PDF
1 Table = 1000 Words? Foundation Models for Tabular Data
PDF
Chunking, Embeddings, and Vector Databases
PDF
Milvus: Scaling Vector Data Solutions for Gen AI
PDF
Advanced Retrieval Augmented Generation Techniques
PDF
09-03-2024_UnstructuredDataAndAIDiscussion.pdf
PDF
Multi-agent Systems with Mistral AI, Milvus and Llama-agents
PDF
Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & ...
PDF
Open Source Milvus Vector Database v 2.6
PDF
Multi-agent Systems with Mistral AI, Milvus and Llama-agents
PDF
MultiModal RAG using vLLM and Pixtral - Stephen Batifol
PDF
MultiModal RAG using vLLM and Pixtral - Stephen Batifol
Supercharge Spark: Unleashing Big Data Potential with Milvus for RAG systems
2025-04-05 - Block71 Event - The Landscape of GenAI and Ecosystem.pdf
Zilliz Cloud Demo for performance and scale
09-19-2024 AI Camp Hybrid Seach - Milvus for Vector Database
Scaling Vector Search: How Milvus Handles Billions+
Building Production Ready Search Pipelines with Spark and Milvus
Multimodal Retrieval-Augmented Generation (RAG) with Vector Database
Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI...
09-25-2024 NJX Venture Summit Introduction to Unstructured Data
1 Table = 1000 Words? Foundation Models for Tabular Data
Chunking, Embeddings, and Vector Databases
Milvus: Scaling Vector Data Solutions for Gen AI
Advanced Retrieval Augmented Generation Techniques
09-03-2024_UnstructuredDataAndAIDiscussion.pdf
Multi-agent Systems with Mistral AI, Milvus and Llama-agents
Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & ...
Open Source Milvus Vector Database v 2.6
Multi-agent Systems with Mistral AI, Milvus and Llama-agents
MultiModal RAG using vLLM and Pixtral - Stephen Batifol
MultiModal RAG using vLLM and Pixtral - Stephen Batifol
Ad

More from Zilliz (20)

PDF
Build Fast, Scale Faster: Milvus vs. Zilliz Cloud for Production-Ready AI
PDF
Zilliz Cloud Monthly Technical Review: May 2025
PDF
Smarter RAG Pipelines: Scaling Search with Milvus and Feast
PDF
Webinar - Zilliz Cloud Monthly Demo - March 2025
PDF
What Makes "Deep Research"? A Dive into AI Agents
PDF
Combining Lexical and Semantic Search with Milvus 2.5
PDF
Bedrock Data Automation (Preview): Simplifying Unstructured Data Processing
PDF
Deploying a Multimodal RAG System Using Open Source Milvus, LlamaIndex, and vLLM
PDF
February Product Demo: Discover the Power of Zilliz Cloud
PDF
Full Text Search with Milvus 2.5 - UD Meetup Berlin Jan 23
PDF
Building the Next-Gen Apps with Multimodal Retrieval using Twelve Labs & Milvus
PDF
Voice-to-Value- LLM-Powered Customer Interaction Analysis.pdf
PDF
Accelerate AI Agents with Multimodal RAG powered by Friendli Endpoints and Mi...
PDF
How Milvus allows you to run Full Text Search
PDF
How to Optimize Your Embedding Model Selection and Development through TDA Cl...
PDF
Keeping Data Fresh: Mastering Updates in Vector Databases
PDF
GraphRAG Agents with Neo4j, Milvus and GPT4
PDF
Using LLM Agents with Llama 3.2, LangGraph and Milvus
PDF
Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!
PDF
Vector Databases for Enhanced Classification
Build Fast, Scale Faster: Milvus vs. Zilliz Cloud for Production-Ready AI
Zilliz Cloud Monthly Technical Review: May 2025
Smarter RAG Pipelines: Scaling Search with Milvus and Feast
Webinar - Zilliz Cloud Monthly Demo - March 2025
What Makes "Deep Research"? A Dive into AI Agents
Combining Lexical and Semantic Search with Milvus 2.5
Bedrock Data Automation (Preview): Simplifying Unstructured Data Processing
Deploying a Multimodal RAG System Using Open Source Milvus, LlamaIndex, and vLLM
February Product Demo: Discover the Power of Zilliz Cloud
Full Text Search with Milvus 2.5 - UD Meetup Berlin Jan 23
Building the Next-Gen Apps with Multimodal Retrieval using Twelve Labs & Milvus
Voice-to-Value- LLM-Powered Customer Interaction Analysis.pdf
Accelerate AI Agents with Multimodal RAG powered by Friendli Endpoints and Mi...
How Milvus allows you to run Full Text Search
How to Optimize Your Embedding Model Selection and Development through TDA Cl...
Keeping Data Fresh: Mastering Updates in Vector Databases
GraphRAG Agents with Neo4j, Milvus and GPT4
Using LLM Agents with Llama 3.2, LangGraph and Milvus
Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!
Vector Databases for Enhanced Classification
Ad

Recently uploaded (20)

PDF
A comparative analysis of optical character recognition models for extracting...
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
PPTX
Tartificialntelligence_presentation.pptx
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PPTX
Programs and apps: productivity, graphics, security and other tools
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
Approach and Philosophy of On baking technology
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PDF
NewMind AI Weekly Chronicles - August'25-Week II
PDF
Machine learning based COVID-19 study performance prediction
PDF
MIND Revenue Release Quarter 2 2025 Press Release
PPTX
Big Data Technologies - Introduction.pptx
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPTX
A Presentation on Artificial Intelligence
A comparative analysis of optical character recognition models for extracting...
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SOPHOS-XG Firewall Administrator PPT.pptx
Tartificialntelligence_presentation.pptx
Mobile App Security Testing_ A Comprehensive Guide.pdf
Programs and apps: productivity, graphics, security and other tools
Dropbox Q2 2025 Financial Results & Investor Presentation
Approach and Philosophy of On baking technology
The Rise and Fall of 3GPP – Time for a Sabbatical?
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
Building Integrated photovoltaic BIPV_UPV.pdf
NewMind AI Weekly Chronicles - August'25-Week II
Machine learning based COVID-19 study performance prediction
MIND Revenue Release Quarter 2 2025 Press Release
Big Data Technologies - Introduction.pptx
MYSQL Presentation for SQL database connectivity
Diabetes mellitus diagnosis method based random forest with bat algorithm
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
20250228 LYD VKU AI Blended-Learning.pptx
A Presentation on Artificial Intelligence

Introduction to Large Language Model Customization.pdf

  • 1. 1 | © Copyright 2024 Zilliz 1 Yujian Tang | Zilliz Introduction to LLM Customization
  • 2. 2 | © Copyright 2024 Zilliz 2 Yujian Tang Senior Developer Advocate, Zilliz yujian@zilliz.com https://www.linkedin.com/in/yujiantang https://www.twitter.com/yujian_tang Speaker
  • 3. 3 | © Copyright 2024 Zilliz 3 01 Introduction to LLMs CONTENTS 03 04 Examples 02 RAG Fine Tuning
  • 4. 4 | © Copyright 2024 Zilliz 4 01 Introduction to LLMs
  • 5. 5 | © Copyright 2024 Zilliz 5 A Basic Neural Net
  • 6. 6 | © Copyright 2024 Zilliz 6 A Recurrent Neural Network
  • 7. 7 | © Copyright 2024 Zilliz 7 A Transformer Architecture
  • 8. 8 | © Copyright 2024 Zilliz 8 GPT
  • 9. 9 | © Copyright 2024 Zilliz 9 02 RAG
  • 10. 10 | © Copyright 2024 Zilliz 10 RAG RAG Inject your data via a vector database like Milvus/Zilliz Query LLM Milvus Your Data Primary Use Case - Factual Recall - Forced Data Injection - Cost Optimization Embed
  • 11. 11 | © Copyright 2024 Zilliz 11 What Does Vector Data Look Like?
  • 12. 12 | © Copyright 2024 Zilliz 12 Find Semantically Similar Data Apple made profits of $97 Billion in 2023 I like to eat apple pie for profit in 2023 Apple’s bottom line increased by record numbers in 2023
  • 13. 13 | © Copyright 2024 Zilliz 13 But wait! There’s more!
  • 14. 14 | © Copyright 2024 Zilliz 14 RAG lets us inject data via semantic similarity provided by vector databases like Milvus
  • 15. 15 | © Copyright 2024 Zilliz 15 03 Fine Tuning
  • 16. 16 | © Copyright 2024 Zilliz 16 RAG vs Fine Tuning LLM Fine Tuning Augment an LLM by training it on your data Your Data “New” LLM Query Primary Use Case - Style transfer - Domain specific usage
  • 17. 17 | © Copyright 2024 Zilliz 17
  • 18. 18 | © Copyright 2024 Zilliz 18
  • 19. 19 | © Copyright 2024 Zilliz 19 Types of Fine Tuning - Full fine tuning - LoRA - QLoRA
  • 20. 20 | © Copyright 2024 Zilliz 20 Fine Tuning Methods - Supervised Fine Tuning - Direct Preference Optimization - 𝚿 (Identity) Preference Optimization - Odds-Ratio Preference Optimization
  • 21. 21 | © Copyright 2024 Zilliz 21 Fine tuning lets us train LLMs to operate in certain styles or domains
  • 22. 22 | © Copyright 2024 Zilliz 22 04 Examples Give Milvus a Star!
  • 23. 23 | © Copyright 2024 Zilliz 23 RAG without OpenAI project
  • 24. 24 | © Copyright 2024 Zilliz 24 AI Agents Projects
  • 25. 25 | © Copyright 2024 Zilliz 25 Fine Tuning Library
  • 26. 26 | © Copyright 2024 Zilliz 26 Start building with Zilliz Cloud today! zilliz.com/cloud