Vector search and multimodal embeddings in BigQuery
0 likes120 views
This document presents an overview of vector search and multimodal embeddings in BigQuery, highlighting their advantages over traditional keyword search methods. It discusses challenges faced by conventional databases, such as scalability and handling multimedia data, while emphasizing the semantic understanding provided by embeddings. The document also outlines practical applications and upcoming features for generating and utilizing embeddings in BigQuery to enhance search capabilities.
![Vector embeddings capturetheunderlyingmeaningof your data
Vector search and multimodal embeddings BigQuery @martonkodok
User
[0.1, 0.002,
-0.56, 0.98...]
App + AI Embeddings
Capture
themeaningofyourdata](https://image.slidesharecdn.com/vectorsearchandmultimodalembeddingsinbigquery-241026134035-a5285a31/85/Vector-search-and-multimodal-embeddings-in-BigQuery-7-320.jpg)
![Vector search and multimodal embeddings BigQuery @martonkodok
We’re no longer confined to text. Generate embeddings on Multimodal data.
Input
[0.1, 0.002,
0.56, 0.98...]
Multimodal
Embedding Model Embeddings
[0.93, 0.133,
0.142, 0.03...]
[0.22, 0.092,
0.391, 0.78...]](https://image.slidesharecdn.com/vectorsearchandmultimodalembeddingsinbigquery-241026134035-a5285a31/85/Vector-search-and-multimodal-embeddings-in-BigQuery-9-320.jpg)
Vector search and multimodal embeddings in BigQuery
- 1. Zagreb, Croatia October 2024 Vector search and multimodal embeddings Márton Kodok Software Architect at REEA.net
- 2. 1. Intro into BigQuery 2. What are multimodal embeddings? 3. Understanding Vector search concepts 4. Vector search in BigQuery 5. Demo 1: SQL syntax 6. Demo 2: Example app about multimodal use cases Agenda Vector search and multimodal embeddings BigQuery @martonkodok
- 3. ● Google Developer Expert on Cloud technologies (2016→) ● Champion of Google Cloud Innovators program (2021→) ● Among the Top 3 romanians on Stackoverflow 207k reputation ● Crafting Cloud Architecture+ML backends at REEA.net Articles: martonkodok.medium.com Twitter: @martonkodok Slideshare:martonkodok StackOverflow: pentium10 GitHub: pentium10 Vector search and multimodal embeddings BigQuery @martonkodok About me
- 5. Vector search and multimodal embeddings BigQuery @martonkodok The keyword search limitations on traditional databases ID Name City 001 Foo NYC 002 Bar LDN User App Tabular data and keywords Searchterm/keyword
- 6. Vector search and multimodal embeddings BigQuery @martonkodok 1. Exact Match Limitations: rely on keyword matching, missing nuances like synonyms or different phrasing. 2. Struggle with Multimedia Data: They are poorly equipped to handle unstructured data types like images, audio, and video. 3. Scalability Challenges: Processing massive datasets of diverse formats becomes increasingly inefficient and slow. The keywordsearch limitations on traditional databases
- 7. Vector embeddings capturetheunderlyingmeaningof your data Vector search and multimodal embeddings BigQuery @martonkodok User [0.1, 0.002, -0.56, 0.98...] App + AI Embeddings Capture themeaningofyourdata
- 8. Vector search and multimodal embeddings BigQuery @martonkodok 1. Semantic Understanding Embeddings capture the meaning and context of data, enabling searches beyond exact matches. 2. Handling Diverse Data Types Embeddings search efficiently handles various data formats, including unstructured data:image/video 3. Results are more relevant and comprehensive, providing a richer user experience. Multimodal embeddings is semantic, not just surface-level keywords
- 9. Vector search and multimodal embeddings BigQuery @martonkodok We’re no longer confined to text. Generate embeddings on Multimodal data. Input [0.1, 0.002, 0.56, 0.98...] Multimodal Embedding Model Embeddings [0.93, 0.133, 0.142, 0.03...] [0.22, 0.092, 0.391, 0.78...]
- 10. Joint Embedding Vector Space Image: “gray tabby cat laying in front of a Christmas tree” Text: size color living ● Embed text, image, and video in the same semantic space with the same dimensionality ● Applications: image/video content search, classification, recommendation Vector search and multimodal embeddings BigQuery @martonkodok Multimodal embeddings space
- 11. “ This opens up new possibilities. For example, you could ask questions like… Vector search and multimodal embeddings BigQuery @martonkodok
- 12. Vector search and multimodal embeddings BigQuery @martonkodok Multimodal embeddings use cases 1. Find me a picture similar to some input text 2. Find images of cats similar to this one, but in a snowy landscape. 3. Maybe you ask for customer support issues that resemble this one, even if the words differ. 4. Find items with two persons 5. or two exact items… 6. Finding product recommendations that understand not just your purchase history, but the styles and aesthetics of the images you've been browsing.
- 13. “ How do we obtain the embeddings? Vector search and multimodal embeddings BigQuery @martonkodok
- 14. Vector search and multimodal embeddings BigQuery @martonkodok Google Cloud - Foundation Models via Vertex AI Gemini 1.5 Flash fastest, most cost-efficient model yet for high volume tasks Gemini 1.5 Pro Multimodal reasoning for longer prompts, 1 million context window Imagen 3 Generate images from Text prompts Multimodal Embeddings Extract semantic information Chirp for Speech to Text Build voice enabled applications
- 16. Joint Embedding Vector Space size color living How do we search? Vector search and multimodal embeddings BigQuery @martonkodok
- 17. Approximate Nearest Neighbor Search Vector search and multimodal embeddings BigQuery @martonkodok embedding_space query_item candidate_items Summary: ● Faster search times ● Scalable to larger datasets ● Slightly reduced recall … 1 2 N Query Search Rank
- 18. Part #3 Vector search in BigQuery
- 19. Vector search and multimodal embeddings BigQuery @martonkodok SQL syntax to generate multimodal embeddings in BigQuery ● Generate embeddings ● ML.GENERATE_EMBEDDING (table) creates embeddings on top of GCS objects defined in an Object Table ● ML.GENERATE_EMBEDDING (sql) create embeddings with SQL syntax on text, images, video ● Store embeddings in BigQuery alongside business data for joins and search
- 20. Search the vector space to find information that is similar to the question. Generate embeddings with AI models. Build vector search index. Search vector space. . . . . . . . . . . . . . . . Vector embeddings Vector search space Text Image Audio Video Code 1. Embeddings 2. Index 3. Search Show me rainbow sweaters! Text Image Audio Video Code Vector search and multimodal embeddings BigQuery @martonkodok
- 21. “ Vector search and multimodal embeddings BigQuery @martonkodok Data Multimodal Embeddings Search Input Vertex AI BigQuery Dataset Results Build Embeddings Table Vector Search
- 22. Demo Part #4
- 23. Architecture options Generate embeddings Raw Data In GCS, BigQuery, or elsewhere Using BigQuery, Vertex AI, or other services Store embeddings In BigQuery Create Vector Index In BigQuery Batch Predictions Using BigQuery SQL Auto-Sync Embeddings To Vertex AI Feature Store 2.0 Online Predictions Using Vertex AI Feature Store 2.0 Vector search and multimodal embeddings BigQuery @martonkodok
- 24. Gen AI … repository @martonkodok
- 25. @martonkodok Code repository GenAI for Google Cloud goo.gle/gen-ai-github
- 26. Extend search Recommendation “Recommend a product for this customer (based on CRM data)” Classification Extract & group entities, e.g. names or places, from a piece of text based on context Info Management Employees can access information across documents, reports, emails with simple search Outlier Detection Identify anomalies or fraudulent activities by comparing clickstream data with historical patterns Vector search and multimodal embeddings BigQuery @martonkodok
- 27. Takeaways New ML.GENERATE_EMBEDDING function to create multimodal vector embeddings New VECTOR_SEARCH function for powerful, managed vector search capabilities Start quickly, with the features you expect from BigQuery Create governed RAG applications with LangChain integration Vector search and multimodal embeddings BigQuery @martonkodok
- 28. Resources BQ Embedding Generation Documentation | Blog | Video BQ Vector Search Documentation | Blog| LangChain Demo Assets BQ LangChain Notebook | Gen AI Repo goo.gle/next24-ana302 Continue your learning journey!
- 29. “ But there is more… Vector search and multimodal embeddings BigQuery @martonkodok
- 30. “ @martonkodok Article about Imagen 3 Function Calling in Gemini Remote Functions in BQ
- 31. Linkedin: @martonkodok Thank you. Q&A. Reea.net - Integrated web solutions driven by creativity to deliver projects. Follow for articles: martonkodok.medium.com Slides available on: slideshare.net/martonkodok