Kyryl Truskovskyi: Training and Serving Open-Sourced Foundational Models (UA)
0 likes113 views
The document discusses training and deploying open-sourced foundational machine learning models, particularly Large Language Models (LLMs) like GPT-4, outlining MLOps stages such as monitoring experiments and data pipelines. It highlights challenges related to compliance, differentiation, and user feedback, and offers insights into improving LLM performance through techniques like fine-tuning and optimized prompts. Additionally, it provides resources for benchmarking and tools to assist in training and serving LLMs effectively.
Kyryl Truskovskyi: Training and Serving Open-Sourced Foundational Models (UA)
- 1. Training & Serving Open-Sourced Foundational Models
- 2. 3 4 Advanced Summary Agenda 2 0 1 2 Intro Context Basics
- 3. Intro 3
- 4. About me Tech Lead at Georgian, Create ML courses. Previously: BorealisAI, Neuro, Ring and Depositphotos Connect on GitHub and Linkedin 4
- 5. About me Tech Lead at Georgian, Create ML courses. Previously: BorealisAI, Neuro, Ring and Depositphotos Connect on GitHub and Linkedin 5
- 6. Context 6
- 10. - Costs - Speed - Privacy - Moat Why? 10
- 11. Why? Compliance challenges - PII aspects - Data sent to a third-party API Over-reliance factor - Upper-bounded by third- party tech - No control over quality 11 Differentiation challenges - Cannot incorporate your product’s user feedback - What’s stopping your competitor from doing the same?
- 12. LLM to keep eye on Close: - GPT4 - PaLM2/Gemini - Claude3 - Mistral 12 Open: - Mistral/Mixtral (7B, 8 x 7B) - Llama 2 (7B, 13B, 70B) - Gemma (2B, 7B) - Flan-T5 Our benchmarking toolkit: LLM-Finetuning-Hub (Training, Inference, Costs, Time)
- 13. Make LLM your LLM Options you need to use: - Prompts - RAG - Fine-Tuning 13 *use all
- 14. Make LLM your LLM Options you need to use: - Prompts - RAG - Fine-Tuning 14 *use all
- 17. Basics 17
- 19. Why not FastAPI? 19 Source: https://madewithml.com/courses/mlops/api/
- 20. TGI from Huggingface 20 Source: https://github.com/huggingface/text-generation-inference
- 24. Advanced 24
- 26. Benchmarking: Best costs 26 Source: https://www.philschmid.de/sagemaker-llama-benchmark
- 27. Benchmarking: Best throughput 27 Source: https://www.philschmid.de/sagemaker-llama-benchmark
- 28. Benchmarking: Best latency 28 Source: https://www.philschmid.de/sagemaker-llama-benchmark
- 29. Best cost = $3.50 & 138m throughput = $4.38 & 37m latency = $32 & 271m 29
- 30. Triton Inference Server 30 Source: https://github.com/triton-inference-server/tensorrtllm_backend, https://github.com/triton-inference-server/vllm_backend, https://github.com/NVIDIA/TensorRT-LLM, https://github.com/triton-inference-server/server, https://github.com/triton-inference-server/python_backend
- 31. Lorax & Multi Adapters 31 Source: https://github.com/predibase/lorax
- 32. Sagemaker & VertexAI 32 Sources: https://www.philschmid.de/sagemaker-llama-llm,, GCP: Serve Open-Source LLMs on Google Cloud
- 33. Async inference 33 Sources: https://www.philschmid.de/sagemaker-huggingface-async-inference, AWS Sagemaker: Asynchronous Inference
- 35. Summary 35
- 38. Thanks “Machine Learning in Production” course 38 aionlineday2024 15% off promo code
- 40. Thanks “Machine Learning in Production” course 40 aionlineday2024 15% off promo code