Efficient Inference and Information Retrieval for Agents: SambaNova + Milvus
0 likes192 views
The document outlines innovations in agentic AI, emphasizing the importance of fast interactivity and sophisticated model management for effective information retrieval. It presents Samanova Systems' integrated offerings, including their cloud platform and AI models, aimed at improving automation and user experience. Moreover, it discusses their unique hardware architecture designed to enhance AI inference speeds significantly.
Efficient Inference and Information Retrieval for Agents: SambaNova + Milvus
- 1. Rachel Bakke Product Manager, SambaNova Systems Efficient Inference and Information Retrieval for Agents: SambaNova + Milvus v 1.0
- 2. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 2 2 Agenda 1. Introduction to Agentic AI 2. What is SambaNova? 3. SambaNova Cloud 4. Zilliz Integration 5. Setting up an Agent
- 3. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Shared under NDA What is Coming Next… New Era of AI Automation These new AI use cases will have: ● Multiple Large LLMs for complex reasoning and decomposition ● Many expert models for task-specific execution ● Non-deterministic & complex chains of model requests These systems need: ✓Blazing fast interactivity ✓Ability to gracefully handle very large inputs ✓High-speed model switching Agentic Systems, which are AI systems designed to autonomously pursue complex goals and workflows with limited direct human supervision.
- 4. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 4 4 Customer: I have a vacation package booked with Premier Airlines for tomorrow. My wife however just got a cold, so I need to cancel the booking. Virtual Agent: Sorry to hear that. Our cancelation policy states that you must cancel at least 48 hours in advance. Customer: No, I bought insurance. Can I speak to an agent to help address this? Customer: I have a vacation package booked with Premier Airlines for tomorrow. My wife however just got a cold, so I need to cancel the booking. Planning Function Calling Generate Response Reflection Virtual Agent: Sorry to hear that. It looks like you bought insurance for your package, and I can process the cancelation for you if you would like to proceed Commonly Deployed Chatbot Agent Service Assistant
- 5. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 5 Speed matters — Where GPT4 falls apart Server Client User Request + Document LLM Call #1 90s Application processes request and generates second request Application processes request and generates third request LLM Call #2 90s … Complex agentic flows 110 mins Single Large LLMs running on GPUs introduce unacceptable latency LLM Call #n 90s 10xGPU latency*# of LLM requests 5 Requests=50 seconds vs. 5
- 6. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 6 6 AI Is Becoming a Commodity
- 7. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 7 7 405B is the Best Open-Source Model 164 85 57 56
- 8. Snapshot Founded by pioneers in AI ▪ Founded in 2017 by industry luminaries and originated at Stanford University ▪ Fully integrated generative AI platform, from 4th generation hardware to pre-trained models ▪ $1B+ funding raised Rodrigo Liang Co-founder & CEO Kunle Olukotun Co-founder & Chief Technologist & Stanford Professor Christopher Ré Co-founder & Stanford Professor Sophisticated, long-term investors SambaNova: Who We Are Lip-Bu Tan Executive Chairman
- 9. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only SN40L: The Best Chip Designed for AI “Ceruleanˮ Architecture-based Reconfigurable Dataflow Unit 1.5 TB High Capacity Memory 5nm TSMC 3-tier Dataflow Memory 1,040 RDU Cores 102B Transistors 64 GB High Bandwidth Memory 520 MB On-Chip Memory 638 TFLOPS (bf16) Cerulean SN40L RDU Generative AI Training and Inference
- 10. 10 The Fastest AI Inference on the Best Models
- 11. 11 SambaNova Nvidia Llama 3.2 1B 2477 304 Llama 3.1 8B 1066 93 Llama 3.1 70B 460 32 Llama 3.1 405B 200 14 Over 10X Faster Tokens/Second/User
- 12. No Number of GPUs Can Achieve RDU Performance 12 12
- 14. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only Andrew Ng’s 4 Design Patterns of Agent Systems Reflection Asking one the same or different LLMs to check the work Tool Use Interacting with external sources like web browsers, calendars, vector databases, etc Planning Decompose complex user questions into a collection of tasks for better and more dynamic execution Multi-Agent Collaboration Agents working together to solve more complex problems “Fast token generation is important. Generating more tokens, even from a lower quality LLM can give good resultsˮ - Andrew Ng
- 15. Menlo Ventures : What’s Really Agentic?
- 16. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 16 16 Hello World from SambaNova Cloud
- 17. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary 17 17 Recently integrated SambaNova’s Cloud and Milvus for developers to: 1. Get started in <10 minutes using these two powerful tools 2. Run a full app utilizing RAG and powered by Milvus Better Together with Zilliz https://github.com/sambanova/ai- starter-kit/tree/main/integrations
- 18. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only Retrieval Augmented Generation Analyst Document Parse and Chunking PDF Upload Vector Embeddings Milvus Query Chunks Retrieval Store Re-Ranker Consumer Similar Vectors Sorted Chunks Response Analysts Upload Knowledge Documents Consumers Retrieve Knowledge
- 19. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 19 19 Run the AI Starter Kit on our Cloud or locally to customize.
- 20. Example of Agent with RAG from our Hackathon 20 https://sambanova.devpost.com/submissions/586146-pokecompanion
- 21. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only 21 21 Questions? Join the Community Community.SambaNova.ai
- 22. Upcoming Events
- 23. Feedback? Come talk to us!
- 24. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only Try It Today cloud.sambanova.ai 24
- 25. Copyright © 2024 SambaNova Systems Inc. | Confidential & Proprietary | Internal Use Only THANK YOU! 25