LLMOps: Match report from the top of the 5th
Download as PPTX, PDF0 likes111 views
The document discusses LLMOps (Large Language Model Operations) compared to traditional MLOps. Some key points: - LLMOps and MLOps face similar challenges across the development lifecycle, but LLMOps requires more GPU resources and integration is faster due to more models in each application. Evaluation is also less clear. - The LLMOps field is around the 5th generation of models, with debates around proprietary vs open source models, and balancing privacy, cost and control. - LLMOps platforms are emerging to provide solutions for tasks like prompting, embedding databases, evaluation, and governance, similar to how MLOps platforms have evolved.
![H2O.ai Confidential
Why:
[Dev|ML|LLM]Ops?](https://image.slidesharecdn.com/13-231120185124-e3c72c91/85/LLMOps-Match-report-from-the-top-of-the-5th-3-320.jpg)
LLMOps: Match report from the top of the 5th
- 1. H2O.ai Confidential Stefan Krawczyk Co-creator of Hamilton CEO & Co-Founder
- 2. H2O.ai Confidential LLMOps: Match report from the top of the 5th Stefan Krawczyk CEO & Co-founder
- 4. H2O.ai Confidential *Ops == Leverage Leverage ⇒ 📈 ROI
- 5. H2O.ai Confidential Talk Overview 🤔 MLOps vs LLMOps 🎙 Top of the 5th 🔮 Forecast 🏠 Take home
- 6. H2O.ai Confidential MLOps vs LLMOps vs
- 7. H2O.ai Confidential MLOps vs LLMOps vs MLOps ~4 years LLMOps < 1 year
- 9. H2O.ai Confidential MLOps Recap vs Idea & Data/Resources
- 10. H2O.ai Confidential MLOps Recap vs Development/ Prototype Idea & Data/Resources
- 11. H2O.ai Confidential MLOps Recap vs Development/ Prototype Getting to Production Idea & Data/Resources
- 12. H2O.ai Confidential MLOps Recap vs Development/ Prototype Getting to Production Maintenance & Business Value Idea & Data/Resources
- 13. H2O.ai Confidential MLOps Recap vs Idea & Data/Resources Development/ Prototype Getting to Production Maintenance & Business Value Design Model Development Operations
- 15. H2O.ai Confidential LLMOps ⊆ MLOps vs Idea & Resources Development/ Prototype Getting to Production Maintenance & Business Value Design Model Development Operations
- 16. H2O.ai Confidential LLMOps ⊆ MLOps vs Prompts Vector Databases Foundation Model Platform / Provider Factuality Assurance Safety Filters Human Evaluation Fine Tuning Embeddings
- 17. H2O.ai Confidential LLMOps ⊆ MLOps vs High level: LLMOps vs MLOps ● Same general shape of problems
- 18. H2O.ai Confidential LLMOps ⊆ MLOps vs High level: LLMOps vs MLOps ● Same general shape of problems Low level: LLMOps vs MLOps ● GPUs required ● Application integration pace ● More “models” in a single application ● Evaluation is fuzzier
- 19. H2O.ai Confidential Top of the 5th
- 20. H2O.ai Confidential Top of the 5th Two teams: ● Proprietary vs Open source Innings: ● around the 5th generation of models Fans: ● Privacy vs Cost vs Controllability
- 21. H2O.ai Confidential Top of the 5th LLMOps Space: ● Point solutions: - prompts - hosting - governance - tracing - embeddings - vector DBs - evaluation - cost tracking - etc MLOps Platforms → LLMOps Platforms
- 22. H2O.ai Confidential A challenging play: GPT3.5 -> GPT4
- 23. H2O.ai Confidential A challenging play: GPT3.5 -> GPT4 Summarize this text from a scientific article. Extract any key points with reasoning. Content:
- 24. H2O.ai Confidential Summarize this text from a scientific article. Extract any key points with reasoning. Content: Same prompt different output! A challenging play: GPT3.5 -> GPT4
- 25. H2O.ai Confidential Other things to think about… - context windows - tokens & cost - latency A challenging play: GPT3.5 -> GPT4
- 26. H2O.ai Confidential Other things to think about… - context windows - tokens & cost - latency Q: is it worth it? A challenging play: GPT3.5 -> GPT4
- 27. H2O.ai Confidential Other things to think about… - context windows - tokens & cost - latency Q: is it worth it? ⇒ 🦙 LLMOps processes ⇒ 🦙 Evaluation is your backbone A challenging play: GPT3.5 -> GPT4
- 28. H2O.ai Confidential Some curveballs seen 1. Not controlling model updates* 2.New foundational model provider features e.g. plugins, privacy, multi-modal, PDF upload
- 30. v H2O.ai Confidential - Cost curves 📉 Computation Forecast
- 31. v H2O.ai Confidential - Cost curves 📉 Computation - Proprietary vs Open Source - Improvements: - Context windows - Simpler prompts - Multi-modal Foundational Models Forecast
- 32. v H2O.ai Confidential - Cost curves 📉 Computation - Proprietary vs Open Source - Improvements: - Context windows - Simpler prompts - Multi-modal Foundational Models - Data is your moat - Evaluations are 🔑 - FE & BE Devs learning LLMOps - Shift to fine-tuning - Still need DS & ML Organizationally Forecast
- 33. v H2O.ai Confidential - Cost curves 📉 Computation - Proprietary vs Open Source - Improvements: - Context windows - Simpler prompts - Multi-modal Foundational Models - Data is your moat - Evaluations are 🔑 - FE & BE Devs learning LLMOps - Shift to fine-tuning - Still need DS & ML Organizationally - Reduction in “wrappers” - Focus on fine tuning & evaluation LLMOps Space Forecast
- 34. H2O.ai Confidential To bring it home: 1. Plan for rapid evolution 2.⚾️ Strong practices to play
- 35. H2O.ai Confidential What I’m building: Standardizing code to enable simpler *Ops. + Hamilton open source
- 36. H2O.ai Confidential Stefan Krawczyk CEO & Co-Founder DAGWorks Inc. stefan@dagworks.io in/skrawczyk Contact Questions? Yes: all images were generated by DALL-E3