Multimodal Graph RAG (mmGraphRAG): Incorporating Vision in Search and Analytics
0 likes1,671 views
The document discusses the multimodal graph RAG (mmgraphrag) framework, which integrates visual and textual data in image search and analytics to enhance search accuracy and interpretability. It emphasizes the importance of addressing challenges posed by siloed data and outlines various use cases such as patent searches, medical imaging, and e-commerce. Additionally, it introduces hyperdimensional computing as a method for effectively handling large-scale multimodal datasets, enhancing the potential for innovative applications.
![(:Agents)-[:ORCHESTRATED_IN]->( )](https://image.slidesharecdn.com/datadaytx-25-250128162507-e8084c5f/85/Multimodal-Graph-RAG-mmGraphRAG-Incorporating-Vision-in-Search-and-Analytics-16-320.jpg)
![ENTERPRISE KNOWLEDGE
Request & Agent Processing
{
“request”:”Find images of bananas on a brown wooden table.”
"query_features": {
"objects": [
"bananas",
"table"
],
"colors": [
"brown"
],
"caption": "A bunch of bananas sitting on top of a brown wooden table."
}](https://image.slidesharecdn.com/datadaytx-25-250128162507-e8084c5f/85/Multimodal-Graph-RAG-mmGraphRAG-Incorporating-Vision-in-Search-and-Analytics-29-320.jpg)
![ENTERPRISE KNOWLEDGE
Vector Similarity Search Results
"results": [
{
"name": "COCO_val2014_000000003661.jpg",
"similarity_score": 0.7162067890167236,
"objects": [
"banana"
],
"colors": [
"dark, vivid yellow"
],
"caption": "A bunch of bananas sitting on top of a wooden table.",
"complexity": "unknown",
"pattern": "none"
}](https://image.slidesharecdn.com/datadaytx-25-250128162507-e8084c5f/85/Multimodal-Graph-RAG-mmGraphRAG-Incorporating-Vision-in-Search-and-Analytics-30-320.jpg)
![ENTERPRISE KNOWLEDGE
Graph Results
"results": [
{
"name": "COCO_val2014_000000003661.jpg",
"similarity_score": 0.7162067890167236,
"objects": [
"medium, muted yellow bowl (LEFT_OF: banana, RIGHT_OF: spoon, ABOVE: spoon, BELOW:
banana, OVERLAPS: banana, ABOVE: bottle, LEFT_OF: bottle)",
"medium, soft yellow banana (LEFT_OF: bowl, RIGHT_OF: spoon, ABOVE: spoon, BELOW: bowl,
OVERLAPS: bowl, OVERLAPS: spoon, OVERLAPS: bottle, ABOVE: bottle, LEFT_OF: bottle)",
"medium, muted yellow spoon (LEFT_OF: bottle, RIGHT_OF: bowl, ABOVE: bowl, OVERLAPS:
banana, ABOVE: banana, ABOVE: bottle, RIGHT_OF: banana)",
"medium, muted yellow bottle (LEFT_OF: bowl, ABOVE: bowl, OVERLAPS: banana, ABOVE:
banana, ABOVE: spoon, LEFT_OF: banana, LEFT_OF: spoon)"
],
"colors": [
"dark, soft yellow"
],
"caption": "A banana that is sitting in a bowl on the table."
},](https://image.slidesharecdn.com/datadaytx-25-250128162507-e8084c5f/85/Multimodal-Graph-RAG-mmGraphRAG-Incorporating-Vision-in-Search-and-Analytics-31-320.jpg)
Multimodal Graph RAG (mmGraphRAG): Incorporating Vision in Search and Analytics
- 1. ENTERPRISE KNOWLEDGE What is next after GraphRAG? Are dinosaurs taking over our bathrooms?
- 2. ENTERPRISE KNOWLEDGE What is the role of images in RAG? A toy dinosaur…
- 3. ENTERPRISE KNOWLEDGE What is the role of images in RAG? A toy dinosaur…
- 4. ENTERPRISE KNOWLEDGE Can multimodal search integrate Semantics and Reasoning? RIGHT_OF A toy dinosaur…
- 5. ENTERPRISE KNOWLEDGE mmGraphRAG Analytics "The most relevant images are 3661 and 1153. Both images feature bananas as the main object, with one image showing a banana in a bowl (3661) and the other image showing a bunch of bananas on top of a table (1153). The colors used are muted yellow, which matches the query's color specification. The spatial relationships between objects are also consistent with the query, with the banana being placed in or next to a bowl or on a table. Both images are bananas sitting on a table, making them the most relevant results."
- 6. Multimodal Graph RAG (mmGraphRAG): Incorporating Vision in Search and Analytics Data Day Texas - 2025 David Hughes Principal Solution Architect Advanced Data Services and Enterprise AI Enterprise Knowledge, LLC. “I research problem solving in complex domains”
- 7. Objectives Demonstrate how mmGraphRAG works. Highlight the integration of vision models, hypervectors, and graph databases. Showcase real-world applications and benefits. Introduce BAML Agentic Workflows.
- 8. ENTERPRISE KNOWLEDGE Why Are We All Here Today? ⬢ Modern systems are overwhelmed by the growing volume of multimodal data. ⬢ Visual and textual data are often siloed, leading to incomplete insights. ⬢ Lack of tools that can explain complex multimodal queries in an intuitive way. ⬢ Reduced search accuracy impacts reputation and user trust. ⬢ Siloed decisions, missing holistic context. ⬢ Delays in retrieving actionable insights due to incomplete or inaccurate search results. ⬢ Inability to identify trends or patterns hidden in multimodal data. ⬢ Vision search without interpretability of the results, xAI. The Challenge Persists What It’s Costing Us
- 9. ENTERPRISE KNOWLEDGE What is mmGraphRAG?
- 10. mmGraphRAG Highlights A framework that combines multimodal data with graph-based RAG for image search and analytics An augmentation of traditional image search that implements Agents and HDC An approach that provides explains results from both visual and textual perspectives Incorporates critical context that image vectors alone cannot capture.
- 11. mmGraphRAG Benefits Search Customization Supports nuanced queries using embeddings, object relationships, colors, and captions. Contextual Understanding Can reason about spatial relationships (e.g., “banana LEFT_OF bowl”) for greater accuracy. Precision in Object Recognition Matches based on exact features, embeddings, and relationships encoded in the graph. Similarity Search Utilizes vector embeddings to find semantically similar images, enabling deeper understanding. Graph Reasoning Explores relationships and hierarchies among objects, colors, and features within the graph. Explainability (xAI) Transparent, interpretable analysis of similarity scores, objects, and spatial features. Customizable Domain Knowledge Tailored graph schema to integrate domain knowledge (e.g., captions, features, object relations). Offline Usability Can operate locally with a pre-built graph and vector database. Privacy & Data Control Operates in a controlled environment, maintaining data privacy and security.
- 12. mmGraphRAG | Aligned Use Cases Intellectual Property and Patent Search Compare new designs or products against existing patents by identifying visual and textual similarities in a privacy-preserving manner. Medical Imaging and Diagnostics Search for images with specific medical conditions or abnormalities (e.g., X-rays or MRI scans) based on features, spatial relationships, and annotations. Education and Research Find images matching specific pedagogical needs or research criteria. Surveillance and Security Identify similar objects or scenarios in surveillance images based on context, spatial relationships, and object features. Design and Creative Industries Assists graphic designers and marketers in finding conceptually similar images based on aesthetics (colors, patterns, and object relationships). Cultural Heritage and Archival Management Search and catalog images from historical archives based on complex visual features and captions. E-Commerce Product Search Enhanced product searches for visually similar items (e.g., “products with a yellow handle”) by leveraging embeddings and relational data. Geospatial Analysis Searching satellite or aerial images for specific features (e.g., “buildings with red roofs near water”). Gaming and Virtual Reality Identify visual assets or textures for game development by matching conceptual queries with image embeddings. Other Associative and Semantic Image Use Cases…
- 13. ENTERPRISE KNOWLEDGE High Level Architecture ● Data Extraction and Embedding ○ Visual features extracted using pretrained models. ○ Features converted into hypervectors for high-dimensional encoding. ● Query and Explain ○ Perform vector similarity searches to identify top similar images. ○ Use graph queries to retrieve contextual details about similar images. ○ Language models generate explanations linking results to user queries. ● Semantic & Graph Construction ○ Load hypervectors and relationships into vector and graph databases. ○ Connect nodes representing image features, metadata, and text-based context. Semantic Layer Data Sources Presentation Layer
- 17. ENTERPRISE KNOWLEDGE What is Hyperdimensional Computing? Sunday Session: Hyperdimensional Horizons: Exploring Neuromorphic Intelligence and Graph Applications We'll explore the principles of HDC, its alignment with brain-inspired architectures, and its transformative potential in graph-baed applications.
- 18. ENTERPRISE KNOWLEDGE Hyperdimensional Computing A computational framework inspired by the brain's way of processing information. Uses high-dimensional vectors (hypervectors) to represent and manipulate data.
- 19. ENTERPRISE KNOWLEDGE Benefits of HDC Handles large-scale multimodal datasets effectively. Scalability Enhances the power of graph queries by embedding rich multimodal relationships. Integration with Graphs Encodes high-dimensional data compactly, enabling faster similarity searches. Efficient Representation Maintains accuracy even with incomplete or noisy data inputs. Robustness to Noise Mimics neural processes in the brain, making it a promising paradigm for neuromorphic and cognitive computing Biologically Inspired
- 20. ENTERPRISE KNOWLEDGE Bundling Binding HDC in Action { "name": "COCO_val2014_000000000208.jpg", "caption": "A toy dinosaur standing on a sink next to a running faucet.", "object": "dinosaur" } caption A toy… object dinosaur
- 21. ENTERPRISE KNOWLEDGE HDC - The Semantic Layer
- 22. ENTERPRISE KNOWLEDGE Semantic Layer | Associative, Local
- 24. ENTERPRISE KNOWLEDGE The Graph Layer
- 26. ENTERPRISE KNOWLEDGE Image Graph | Discrete, Global
- 29. ENTERPRISE KNOWLEDGE Request & Agent Processing { “request”:”Find images of bananas on a brown wooden table.” "query_features": { "objects": [ "bananas", "table" ], "colors": [ "brown" ], "caption": "A bunch of bananas sitting on top of a brown wooden table." }
- 30. ENTERPRISE KNOWLEDGE Vector Similarity Search Results "results": [ { "name": "COCO_val2014_000000003661.jpg", "similarity_score": 0.7162067890167236, "objects": [ "banana" ], "colors": [ "dark, vivid yellow" ], "caption": "A bunch of bananas sitting on top of a wooden table.", "complexity": "unknown", "pattern": "none" }
- 31. ENTERPRISE KNOWLEDGE Graph Results "results": [ { "name": "COCO_val2014_000000003661.jpg", "similarity_score": 0.7162067890167236, "objects": [ "medium, muted yellow bowl (LEFT_OF: banana, RIGHT_OF: spoon, ABOVE: spoon, BELOW: banana, OVERLAPS: banana, ABOVE: bottle, LEFT_OF: bottle)", "medium, soft yellow banana (LEFT_OF: bowl, RIGHT_OF: spoon, ABOVE: spoon, BELOW: bowl, OVERLAPS: bowl, OVERLAPS: spoon, OVERLAPS: bottle, ABOVE: bottle, LEFT_OF: bottle)", "medium, muted yellow spoon (LEFT_OF: bottle, RIGHT_OF: bowl, ABOVE: bowl, OVERLAPS: banana, ABOVE: banana, ABOVE: bottle, RIGHT_OF: banana)", "medium, muted yellow bottle (LEFT_OF: bowl, ABOVE: bowl, OVERLAPS: banana, ABOVE: banana, ABOVE: spoon, LEFT_OF: banana, LEFT_OF: spoon)" ], "colors": [ "dark, soft yellow" ], "caption": "A banana that is sitting in a bowl on the table." },
- 32. ENTERPRISE KNOWLEDGE Graph Results "analysis": "The most relevant images are COCO_val2014_000000003661 and COCO_val2014_000000001153. Both images feature bananas as the main object, with one image showing a banana in a bowl (COCO_val2014_000000003661) and the other image showing a bunch of bananas on top of a table (COCO_val2014_000000001153). The colors used are muted yellow, which matches the query's color specification. The spatial relationships between objects are also consistent with the query, with the banana being placed in or next to a bowl or on a table. The captions for both images describe bananas sitting on a table, making them the most relevant results."
- 34. BrainGraph | A Different Use Case for Image Data Medical images consist of 3D pixels called voxels. Voxels are nodes in a graph with neighbors. Communities of voxels can represent anatomical structures, or abnormalities like tumors. Evolution in the graph can represent disease progression or treatment response.