SaaStr Annual 2024: Outsmarting LLMs: 5 Strategies for Founders & Technologists with Zencoder
Download as PPTX, PDF0 likes54 views
The document discusses the limitations and failure modes of large language models (LLMs), including issues with multi-hop reasoning and probabilistic memorization. It highlights that significant improvements can be made to LLMs without extensive retraining, through techniques like enhanced context length and tool usage. The text emphasizes the advancing capabilities of open-source LLMs, which are narrowing the gap with proprietary models, encouraging the use of proprietary data and feedback to optimize performance.
SaaStr Annual 2024: Outsmarting LLMs: 5 Strategies for Founders & Technologists with Zencoder
- 1. Outsmarting LLMs: 5 Strategies for Founders & Technologists Andrew Filev CEO & Founder Zencoder AI
- 3. The scale/pre-training has been core driver of LLM performance up until 2024 Performance Time
- 4. And then “the frontier” slowed down a bit Performance Time We are here
- 5. NOT TODAY
- 6. Let’s talk about limitations of LLMs. They are trained to predict the most probable next symbol, even if it sometimes leads to inconsistencies or mistakes.
- 7. Failure modes 1. “Fuzzy” (probabilistic) memorization, somewhat “associative” memory. 3. Struggle with “multi-hop” reasoning, unless extensively trained with similar examples. 2. Not reliable in following an algorithm, unless extensively trained on it. 4. No differentiation in epistemic vs aleatoric uncertainty, no estimates for the cost of error, no explicit evaluation /feedback. 5. They work well “in distribution”. And they are mostly trained on open internet data, which limits that distribution.
- 8. There are ways to combat these failure modes. And it’s not what frontier labs made you believe.
- 9. Performance that doesn’t rely on access to vast computing resources Image: Epoch AI Source: Epoch AI “ AI Capabilities Can Be Significantl y Improved Without Expensive Re training ” “While scaling compute for training is key to improving LLM performance, some post-training enhancements can offer gains equivalent to training with 5 to 20x more compute at less than 1% the cost.
- 10. The Power of Scaling for Reasoning. Exp or Log? Activity Time Pre-training New techniques We are here
- 11. The Future is Still Amazing, and it’s yours! We are still on the exponential curve, but the direction shifted: there have been recent breakthroughs in context length, inference speed, costs. Mar’23 Nov’23 May’24 GPT4 announced GPT-4 Turbo is 3x cheaper for input and 2x cheaper for output GPT-4o is 2x cheaper than GPT-4 Turbo Example:
- 12. 1. “Fuzzy” (probabilistic) memorization, somewhat “associative” memory. 1. Context is key 3. Struggle with “multi-hop” reasoning, unless extensively trained with similar examples. 3. Use various ways to decompose the problem 5 Ways 2. Not reliable in following an algorithm, unless extensively trained on it. 4. No differentiation in epistemic vs aleatoric uncertainty, no estimates for the cost of error, no explicit evaluation /feedback. 5. They work well “in distribution”. And they are mostly trained on open internet data, which limits that distribution. 2. Leverage tools 4. Incorporate feedback in your workflows 5. Experiment with fine tuning on your proprietary data
- 13. The gap between commercial and open- source LLMs is collapsing very fast.
- 14. General comparison of commercial and open-source LLMs Proprietary Open Source MMLU-Pro: the more recent version of a Massive Multitask Language Understanding benchmark
- 15. Your Secret Weapon: Agentic design patterns Planning The LLM comes up with, and executes, a multistep plan to achieve a goal (for example, writing an outline for an essay, then doing online research, then writing a draft, and so on). Tool Use The LLM is given tools such as web search, code execution, or any other function to help it gather information, take action, or process data. Reflection The LLM examines its own work to come up with ways to improve it. Multi-agent collaboration More than one AI agent work together, splitting up tasks and discussing and debating ideas, to come up with better solutions than a single agent would.
- 16. Case Study Lets leverage embedded AI agents to improve code generation. By embedding agentic AI into code assistants we offload routine tasks, improve code coverage and integrate into your development workflows seamlessly.
- 17. Robust AI pipeline can improve automatic code generation LLM Assistant Engineer Missing Context Needed, Irrelevant Context Hallucinations, Incorrect Code LLM Agentic Improvement Assistant Intelligent Context Repairs and Improves Code Engineer AI Agent Current Solution: Typical AI Coding Assistant � � � � Repo Grokking™
- 18. Hypothetical performance in your specific domain Open Source Improved Context 1. Context is key Recent rapid expansion of context length allows significantly improve the reasoning capabilities in complicated scenarios
- 19. Hypothetical performance in your specific domain 2. Leverage tools Tooling Open Source Improved Context
- 20. Hypothetical performance in your specific domain 3. Use various ways to decompose the problem. (Fixed Chain, CoT, Agentic Planning) Tooling Workflow Open Source Improved Context
- 21. 4. Incorporate feedback in your workflows Hypothetical performance in your specific domain Tooling Workflow Feedback Open Source Improved Context
- 22. Hypothetical performance in your specific domain 5. Experiment with fine tuning on your proprietary data Tooling Workflow Feedback Fine-tuning Open Source Improved Context
- 23. Key Takeaways LLMs are incredibly powerful, but that power has its limitations. Open source capabilities are rapidly reaching those of frontier models, giving you a strong foundation to work with. You can address those limitations with your proprietary data, tooling, and know- how, building powerful agentic systems. Increases in context length, inference cost and latency make “in context learning” a viable alternative to training the models, that’s especially important during initial prototyping.
- 24. Q& A