"Scaling RAG Applications to serve millions of users", Kevin Goedecke
0 likes259 views
The document provides insights on scaling retrieval-augmented generation (RAG) applications and the challenges encountered in managing large language models (LLMs) and vector databases. It highlights the lessons learned from the company's growth, including issues with storing vector data, rate limits, and query optimization. The presentation outlines strategies for enhancing performance, such as using Azure for load balancing and implementing better indexing methods for vector data.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
- 1. Scaling RAG applications to serve millions of users Lessons learned scaling LLMs, Vector databases and more… Powered by
- 2. What we’ll cover today… 📝 Agenda - Who is SlideSpeak? What do we do? - Our Growth so far - Mistakes we’ve made - Architecture Overview - Challenges of scaling RAG applications - Challenges of scaling Vector databases - What we’re currently busy with… - Q&A
- 3. Create presentations from Word, PDF or Excel files Create presentations for any topic Some of our key features: SlideSpeak is an AI-powered platform to create and automate presentations What we do
- 4. We save people hours of work by speeding up manual presentation workflows
- 5. OUR GROWTH SO FAR We’ve launched 6 months ago, here’s what we’ve achieved. >2m Files upload >2m LLM Tokens consumed per minute >1000 LLM calls per minute 250k MAUs
- 6. User Vector DB Context LLM Response Query Inference Retrieval Prompt Quick recap on RAG
- 7. Using LLMs for everything Storing vector data forever No monitoring Downtimes cost us thousands of $$$ Mistakes we’ve made… so far… �� �� �� Storing vectors is expensive, like really expensive… If not absolutely necessary use avoid using LLMs
- 8. Architecture
- 9. Prompts, JSON Mode, Regression Scaling Vector Databases Scaling LLM providers Rate limits, Downtimes, … Challenges scaling RAG Applications �� �� �� Slow queries, index building, … LLM specific prompts, Pydantic being unreliable, …
- 10. - Rate limits are not as high as they seem - Difficult to balance RPM and TPM - 40 page document has on avg. 24k tokens, with 2m limit thats 83 documents per minute or 1.5 per second OpenAI Rate Limits (6/10/24) Scaling LLM providers The problem
- 11. - We’ve migrated to Azure OpenAI - Not because the rate limits are higher, but you can load balance 🤯 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models Scaling LLM providers The solution
- 12. Source: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/smart-load-balancing-for-openai-end points-and-azure-api/ba-p/3991616 Scaling LLM providers Load Balancing in Azure
- 14. - Find similar information in a lot of data - Feed that to the LLM as context Why do we need Vector Databases?
- 15. 5 Problems when scaling Postgres with PGVector - 🦥 Slow queries - 🧠 Memory intense - 🔍 Challenge of combining vector search and metadata search (also called hybrid search) needs careful query optimization Challenges of scaling Vector Databases
- 16. Challenges of scaling Vector Databases A Vector is an array of 4-byte floating point numbers. Number of Vectors Total Size 7 Thousand 7,000 × 1536 × 4 = 43 Megabyte 1 Million 1,000,000 × 1536 × 4 = 5.7 Gigabyte 10 Millions 10,000,000 × 1536 × 4 = 57 Gigabyte 1 Billion 1000,000,000 × 1536 × 4 = 5.5 Terabyte
- 17. Build an index over the vector data Challenges of scaling Vector Databases HNSW: Efficient nearest neighbor search algorithm (ANN) for high-dimensional data. ef_construction: Defines how many similarity candidates to look for m: Defines how many of the closest neighbors to pick from the ef_construction list ** This might be more tricky if you use hybrid search (metadata + vector data)
- 18. Define what could be a good partition value for you (date, filetype, category, …) Use partitioning What we’ve done to scale PGVector
- 19. General tips when working with PGVector Make sure to delete unnecessary vector data 01 Never use the same Postgres database for Vectors and other data 02
- 20. If nothing helps… Scale horizontally What we’ve done to scale PGVector
- 21. What we’re currently busy with… System Extension Azure AI Postgres Extension RAG Evaluation – Implementing robust evaluation testing methods – Exploring Ragas for advanced system performance ✅ �� �� – Enabling direct creation of embeddings within Azure – Extend system to cover images and other non-textual data Here’s what currently keeps us up at night…
- 22. Questions? And now it’s your turn…