Agentic Techniques in Retrieval-Augmented Generation with Azure AI Search
Discover how Agentic Retrieval in Azure AI Search takes Retrieval-Augmented Generation (RAG) to the next level by intelligently breaking down complex queries, leveraging full conversation history, and executing parallel searches through a new LLM-powered query planner. This session introduces a cutting-edge approach that delivers significantly more accurate, relevant, and grounded answers—unlocking new capabilities for building smarter, more responsive generative AI applications. Traditional Retrieval-Augmented Generation (RAG) pipelines work well for simple queries—but when users ask complex, multi-part questions or refer to previous conversation history, they often fall short. That’s where Agentic Retrieval comes in: a game-changing advancement in Azure AI Search that brings LLM-powered reasoning directly into the retrieval layer. This session unveils how agentic techniques elevate your RAG-based applications by introducing intelligent query planning, subquery decomposition, parallel execution, and result merging—all orchestrated by a new Knowledge Agent. You’ll learn how this approach significantly boosts relevance, groundedness, and answer quality, especially for sophisticated enterprise use cases. Key takeaways: - Understand the evolution from keyword and vector search to agentic query orchestration - See how full conversation context improves retrieval accuracy - Explore measurable improvements in answer relevance and completeness (up to 40% gains!) - Get hands-on guidance on integrating Agentic Retrieval with Azure AI Foundry and SDKs - Discover how to build scalable, AI-first applications powered by this new paradigm Whether you're building intelligent copilots, enterprise Q&A bots, or AI-driven search solutions, this session will equip you with the tools and patterns to push beyond traditional RAG.
![RAG: Retrieval Augmented Generation
User
Question
What is included in
my Northwind
Health Plus plan
that is not in
standard?
[Benefit_Options.pdf#page=3]
Both plans offer coverage for
medical services. Northwind
Health Plus offers coverage for
hospital stays, doctor visits, lab
tests, and X-rays. Northwind
Standard only offers coverage
for doctor visits and lab tests. …
Large
Language
Model
Both plans offer coverage for
medical services, such as doctor
visits and lab tests, but the
Health Plus plan also covers
hospital stays and X-rays.1
Search](https://image.slidesharecdn.com/agentictechniquesinretrieval-augmentedgenerationwithazureaisearch-maximsalnikov-250607120225-ef2eeda7/85/Agentic-Techniques-in-Retrieval-Augmented-Generation-with-Azure-AI-Search-6-320.jpg)
Agentic Techniques in Retrieval-Augmented Generation with Azure AI Search
- 1. Agentic Techniques in Retrieval-Augmented Generation with Azure AI Search Maxim Salnikov Digital and App Innovation Business Lead at Microsoft
- 2. I’m Maxim Salnikov Building on web platform since 90s Organizing developer communities and technical conferences Speaking, training, blogging: Webdev, Cloud, Generative AI, Prompt Engineering Helping developers to succeed with the Dev Tools, Cloud & AI in Microsoft
- 3. How to build high-quality search in LLM-infused apps
- 4. Well-known limitations of LLMS Outdated public knowledge No internal knowledge
- 5. Incorporating domain knowledge Fine tuning Learn new skills (permanently) High cost, time Retrieval augmentation Learn new facts (temporarily)
- 6. RAG: Retrieval Augmented Generation User Question What is included in my Northwind Health Plus plan that is not in standard? [Benefit_Options.pdf#page=3] Both plans offer coverage for medical services. Northwind Health Plus offers coverage for hospital stays, doctor visits, lab tests, and X-rays. Northwind Standard only offers coverage for doctor visits and lab tests. … Large Language Model Both plans offer coverage for medical services, such as doctor visits and lab tests, but the Health Plus plan also covers hospital stays and X-rays.1 Search
- 7. Components of a high-quality RAG 1 Sophisticated LLM Adheres to instructions Supports function calling 2 Well prepared data Reasonably sized text Meaningful vectors 3 Powerful search functionality Vector search Hybrid search Semantic re-ranking Filtering
- 8. Azure AI Search State-of-the-art RAG Rigorously tested retrieval Streamlined data pipeline Developer integrations Enterprise-ready foundation
- 9. Streamlined RAG data pipeline Ingest • OneLake • Blob Storage • ADLSv2 • SQL DB • CosmosDB + Incremental change tracking Extract • PDFs, Office docs • Image files • Nested images in PDFs, Office docs • Document layout JSON, CSVs, markdown parsing Chunk • Split text into passages (for text portions): • Fixed-size • AI Document Intelligence layout output • Propagate document metadata Embed • Image → vector • Text → vector • Embeddings from OpenAI, AI Vision, or custom models Index • Document index • Chunk index • Both Query • Boosting • Weighting • Thresholding
- 10. Azure AI Foundry Foundry Models Foundry Agent Service Azure AI Search Azure AI Services Azure Machine Learning Azure AI Content Safety Foundry Observability Security • Identity • Management Copilot Studio Visual Studio GitHub Foundry SDK Cloud Azure Azure Arc Foundry Local Edge Serverless Control Azure Kubernetes Service Azure Container Apps Azure App Service Azure Functions
- 11. Classic search workflow User query L1 L2 → LLM Results as-is from L1 Single-shot retrieval Reranking e.g. cross-encoder Linear ranked list Hybrid RRF(keyword, vector) No interpretation Vector search embeddings + vector index Keyword search segmentation + inverted index
- 12. A complete stack gives you optimal retrieval Question: "What underwater activities can I do in the Bahamas?" Keyword results 1 Scuba Diving in Bahamas Vector results 1 Scuba Diving in the Carribean 2 Water skiing in Seychelles Fusion (RRF) 1 Scuba Diving in the Carribean 2 Scuba Diving in Bahamas 3 Water skiing in Seychelles Reranking with cutoff 1 Scuba Diving in Bahamas 2 Scuba Diving in the Carribean
- 13. RAG can struggle with complex queries Works security updates KB4048959 Doesn’t work what does KB4048959 fix and what systems is the patch compatibel with? copay cost whats the difference in costs for copays vs split pays
- 14. Introducing agentic retrieval in Azure AI Search
- 15. Announcing Agentic retrieval Agentic methods applied to retrieval Query planning Parallel query execution Results merging User query Conversation turns Query planning Search query 1 L1 L2 … L1 L2 Search query n L1 L2 Merged results Single LLM call Use conversation history for context • Correct spellings in context • Decompose queries as needed • Paraphrase queries
- 16. Query planning & decomposition what does KB4048959 fix and what systems is the patch compatible with? Correcting spelling is much more effective if done in sentence context Use chat history for context (the user said earlier it was a security fix) Split queries for different information requirements, resolving co-references
- 17. What is agentic retrieval in Azure AI Search? What is included in my Northwind Health Plus plan that is not in standard? Both plans offer coverage for medical services, such as doctor visits and lab tests, but the Health Plus plan also covers hospital stays and X-rays1 User Question Large Language Model Search Northwind Health Plus plan benefits Search Standard Northwind Health plan benefits Large Language Model
- 18. Introducing Knowledge Agent What is included in my Northwind Health Plus plan that is not in standard? User Question Knowledge Agent Northwind Health Plus plan benefits Standard Northwind Health plan benefits Search Index Activity Log Query Results Northwind Health Plus plan benefits 3 Standard Northwind Health plan benefits 3 Document References Northwind Health Plus Summary.pdf Northwind Health Comparison.pdf Northwind Health Standard Summary.pdf
- 19. Query planning in agentic retrieval User Question What is included in my Northwind Health Plus plan that is not in standard? LLM Reply Northwind Health Plus adds several types of coverage that the Standard plan does not have… User Question How much more does Northwind Health Plus cost? LLM Query Planning Payroll deduction Northwind Health Plus Payroll deduction Northwind Health Standard
- 20. Activity log in agentic retrieval LLM Query Planning Input Tokens: 200 Output Tokens: 50 Search Northwind Health Plus plan benefits Results: 3 Duration: 200ms Search Standard Northwind Health plan benefits Results: 3 Duration: 250ms
- 21. How do we know if it’s actually better?
- 22. How do we know if it’s actually better? Query Answer relevance Is the answer relevant to the query? Context relevance Is the retrieved context relevant to the query? Response Context Groundedness Is the response supported by the context? Source: The RAG Triad | www.trulens.org/getting_started/core_concepts/rag_triad/
- 23. Agentic retrieval evals +40% RAG answer relevance +30% RAG answer result rate For complex queries requiring information from multiple documents 0 10 20 30 40 50 60 Answer score Content score Traditional search vs agentic retrieval Complex Agentric Retrieval Complex Search Full evals results and methodology description: aka.ms/aisearch-arevals
- 24. Resources and next steps Get Started with Azure AI Search aka.ms/AISearch-new Agentic retrieval free in first phase of public preview aka.ms/AgentRAG GitHub Sample – Quickstart aka.ms/AISearch-ar-pyn GitHub Sample – Agent Service Integration aka.ms/AISearch-ar-agent Take the Azure AI Learn Courses aka.ms/CreateAgenticAISolutions
- 25. Thank you! Connect with me on LinkedIn: • Message me to get a link to this slidedeck with all links • Follow me to get the latest AI announcements • Invite me to deliver a technical session or training on Gen AI, Agents, RAG for your company or conference