Vector Databases and Why Are They Used in Modern AI - Marko Lohert - ATD 2024
Download as PPTX, PDF0 likes97 views
Vector databases store and manage high-dimensional vector data, enabling similarity searches based on semantic meaning rather than exact matches. They play a vital role in modern AI applications, such as semantic search and recommendation systems, by quickly retrieving complex data. Popular vector databases include Chroma, Pinecone, Qdrant, and Weaviate, each with unique advantages and use cases.
![Basic idea behind a vector database
• For every piece of data (text, image, ... ) a vector is stored
in vector database
• „Apple” -> [0.52, 0.56, 0.80, … 0.83]
• These vectors have a fixed number of dimensions
• Similar data is clustered together
• Find similar data in database == find similar vectors](https://image.slidesharecdn.com/vectordatabasesandwhyaretheyusedinmodernai-markolohert-atd2024-241202223424-b3153af9/85/Vector-Databases-and-Why-Are-They-Used-in-Modern-AI-Marko-Lohert-ATD-2024-7-320.jpg)
Vector Databases and Why Are They Used in Modern AI - Marko Lohert - ATD 2024
- 2. Vector Databases and Why Are They Used in Modern AI Marko Lohert Span
- 3. Agenda What is a vector database? Popular vector databases Why are vector databases used in modern AI?
- 4. What is a vector database? Popular vector databases Why are vector databases used in modern AI?
- 5. Why do we need a vector database? • In some applications we need • To find similar data (not the exact match) • Semantic search – content aware search, search based on meaning • … And we need to find data fast
- 6. Vector database stores and manages high-dimensional vector data and allows similarity search.
- 7. Basic idea behind a vector database • For every piece of data (text, image, ... ) a vector is stored in vector database • „Apple” -> [0.52, 0.56, 0.80, … 0.83] • These vectors have a fixed number of dimensions • Similar data is clustered together • Find similar data in database == find similar vectors
- 8. Data -> vector apple Apple orange Raspberry Pi 5 is_fruit 1 0 1 0 round 0.86 0 0.93 0 expensive 0.01 9.97 0.02 0.74 use_electricity 0 0.97 0 0.12 company_type 0 0.9 0 0.75 related_to_computers 0 1 0 1 related_to_phone 0 1 0 0 vectors
- 9. Data represented as vectors • Grouping by similarity • Hundreds / thousands of dimensions Image source: https://weaviate.io/blog/what-is-a-vector-database
- 10. Embedding • Embedding == an array of numbers that is used to describe the data • Data can be text (word, sentence), image .. • Often words „embedding” and „vector” are used interchangeable • Note: • word „vector” puts focus on numerical representation itself (mathematics), and word „embedding” puts focus on representing data stored in database
- 11. Embedding • A vector database stores embeddings (+ data) • Embeddings are used to search for (similar) data • Embeddings capture semantic meaning and context similar data have embeddings that are near each other in high- dimensional vector space
- 12. How to calculate an embedding from an input data? • Embedding model uses machine learning models to turn input text, image… into vector / embedding Image source: https://medium.com/@junxie2/vector-in-machine-learning-a24880e3b96a Embedding model
- 13. Embedding models • OpenAI • text-embedding-ada-002 (1536 dimensions) • text-embedding-3-small (1536 dimensions) • text-embedding-3-large (3072 dimensions) • Google Gemini • Multiple embedding models • Ollama Embeddings • Multiple embedding models • Hugging Face • …
- 14. Types of embeddings • Word embeddings • Individual words as vectors • Used techniques: Word2Vec, GloVe, FastText, … • Sentence embeddings • Entire sentences as vectors • Used models: Universal Sentence Encoder (USE), SkipThought, … • Document embeddings • Document as vector (from articles to books) • Used techniques: Doc2Vec and Paragraph Vectors
- 15. Types of embeddings • Image embeddings • Images as vectors (capture different visual features) • Used techniques convolutional neural networks (CNNs), pre- trained models like ResNet and VGG generate image embeddings • Used for image classification, object detection, and finding similar images • …
- 16. How to find similar vectors? • Once we have vectors, generated by embedding models, we can search for similar vectors • Similar vectors data with similar meaning Image source: https://weaviate.io/blog/what-is-a-vector-database
- 17. How to find similar vectors • Multiple methods for calculating distances • Distance between vectors vs similarity between vectors • Let’s take a look at 3 of those methods for calculating similarity …
- 18. Euclidean Distance • Squared Euclidean distance (L2-Squared distance) measures the distance between two vectors x and y: distance =
- 19. Manhattan distance • Manhattan distance (L1 norm or Taxicab Distance) measures the distance between two vectors x and y distance =
- 20. Cosine similarity • Measures the angle between two vectors =
- 21. Cosine similarity • Measures the similarity between documents regardless of the magnitude • One of the most used similarity measurements • Commonly used in Natural Language Processing (NLP)
- 22. Embeddings Index • Vector databases can contain a huge number of vectors • For faster search vectors need to be indexed • There are different ways for calculating indexes • Inverted File (IVF), Navigable Small World (NSW), …
- 23. Different approaches to implementing vector databases 1. Fully vectorized database (Stand-alone) • Example: Chroma, Pinecone … 2. Extensions for traditional databases • For example: pgvector, an open-source extension for PostgreSQL (https://github.com/pgvector)
- 24. Advantages of vector databases • Speed / performance • various indexing techniques for faster searching • Scalability • Flexibility • Vector databases can handle higher complexity of using images, videos or other multidimensional data. • Semantic search
- 25. Disadvantages of vector databases • Because vector databases provide approximate results, applications requiring greater accuracy will need a different database
- 26. Use cases for vector databases • Semantic search • Search based on the meaning or context • Recommendation engine • Web shop recommends similar items to items user has previously bought (in vector database find nearest neighbor of previously bought items) • Natural-language search • Artificial Intelligence / Enhancing LLM capabilities
- 27. Database comparison Relational database • Data is in tables – rows and columns • Looks for the exact match select productName from Product where productId = xxxxx • Highly versatile and can be used for many different applications Vector database • Data represented as vectors • Looks for similar data • Finds products similar to apple
- 29. What is a vector database? Popular vector databases Why are vector databases used in modern AI?
- 30. Chroma • https://www.trychroma.com • A lightweight vector database • Used in RAG applications • Open source - license: Apache 2.0 • GitHub: https://github.com/chroma-core/chroma
- 31. Chroma • Default embedding model: all-MiniLM-L6-v2 • 384 dimensions • Created by Sentence Transformers (https://www.sbert.net) • More info at Hugging Face: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
- 32. Install Chroma Python pip install chromadb chroma run JavaScript pip install chromadb chroma run # install the JS client and the (optional) default embedding function npm install --save chromadb chromadb-default- embed # set "type": "module" in your package.json
- 34. • https://www.pinecone.io • Used for recommendation systems and search engines • Multiple embedding model supported • https://docs.pinecone.io/models/overview
- 35. • Pinecone serverless on Azure, AWS, and Google Cloud Platform • Pricing plan: • Starter – for trying out and for small applications (free) • Standard – for production applications at any scale (paid) • Enterprise – for mission-critical production applications (paid)
- 36. • https://qdrant.tech • Multiple cloud providers: Azure, AWS, Google Cloud Platform • Pricing plan • Managed Cloud (free) • Hybrid Cloud (starting from $0.014 per hour) • Custom (price on request)
- 37. • https://weaviate.io • Open source • Integrates semantic search and knowledge graphs -> weaviate is used for natural language processing (NLP)
- 38. • Multiple cloud providers: Azure, AWS, Google Cloud Platform • Free trial • Pricing plan • Serverless Cloud (starting at $25 per month) • Enterprise Cloud • Bring Your Own Cloud
- 39. What is a vector database? Popular vector databases Why are vector databases used in modern AI?
- 40. Why are vector databases used in modern AI / LLM? • Vector databases deliver performance for GenAI • Vector databases can handle large amounts of complex data • Vector database speed up the training of models because they enable faster data retravel • Long-term memory for LLM-s
- 41. Vector Databases and RAG • RAG (Retrieval-Augmented Generation) • Enables LLMs to use external data (our company data) • Allows LLMs to access the current (up-to-date) data • RAG needs fast access to data, and vector databases are effective at storing, indexing and retrieving that data • Vector databases are essential for RAG applications
- 42. Vector search in Azure AI Search • More info: https://learn.microsoft.com/en-us/azure/search/vector-search-overvi ew
- 43. Key messages • Vector database stores and manages high-dimensional vector data and allows similarity search. • Vector databases: Chroma, Pinecone, qdrant, weaviate… • Vector databases + modern AI: • Performance • Vector databases are essential for RAG applications • Long-term memory for LLM-s • …
Editor's Notes
- #6: Vector Database Explained | What is Vector Database? Semantic search means not searching using the exact keyword matching, but understang the intend of user query and using the context to perform the search Today, our world is increasingly dominated by complex, unstructured data like images, audio, video, and text. Traditional ways of retrieving data based on keyword matching are no longer sufficient. Vector databases are designed to handle complex high-dimensional data, unlocking the foundation for pivotal AI applications.
- #7: What Is A Vector Database? | IBM A vector database stores, manages and indexes high-dimensional vector data. Data points are stored as arrays of numbers called “vectors,” which are clustered based on similarity.
- #11: Vector Databases simply explained! (Embeddings & Indexes) What is a Vector Database & How Does it Work? Use Cases + Examples | Pinecone Index in vector database Embedding is a list of numbers that represents data – a numerical representation of a text Vector embeddings - OpenAI API What are embeddings? OpenAI’s text embeddings measure the relatedness of text strings. Embeddings are commonly used for: Search (where results are ranked by relevance to a query string) Clustering (where text strings are grouped by similarity) Recommendations (where items with related text strings are recommended) Anomaly detection (where outliers with little relatedness are identified) Diversity measurement (where similarity distributions are analyzed) Classification (where text strings are classified by their most similar label) An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness.
- #12: Vector Databases simply explained! (Embeddings & Indexes) What is a Vector Database & How Does it Work? Use Cases + Examples | Pinecone Index in vector database Embedding is a list of numbers that represents data – a numerical representation of a text Vector embeddings - OpenAI API What are embeddings? OpenAI’s text embeddings measure the relatedness of text strings. Embeddings are commonly used for: Search (where results are ranked by relevance to a query string) Clustering (where text strings are grouped by similarity) Recommendations (where items with related text strings are recommended) Anomaly detection (where outliers with little relatedness are identified) Diversity measurement (where similarity distributions are analyzed) Classification (where text strings are classified by their most similar label) An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness.
- #13: https://docs.trychroma.com/integrations/openai What are Vector Embeddings? | A Comprehensive Vector Embeddings Guide | Elastic
- #17: Nearest-neighbour search
- #18: distance = 1 - similarity
- #19: How Vector Databases Search by Similarity: A Comprehensive Primer | by Neil Kanungo | KX Systems | Medium When calculating Euclidean Distance in spaces higher than 2- or 3- dimensions, it i’s often referred to as the “L2-norm.” When comparing embeddings: distance =
- #21: Distance Metrics in Vector Search https://weaviate.io/blog/distance-metrics-in-vector-search Cosine Similarity measures the similarity between documents regardless of the magnitude. This is advantageous because if two documents are far apart by the euclidean distance, the angle between them could still be small. For example, if the word ‘fruit' appears 30 times in one document and 10 in the other, that is a clear difference in magnitude, but the documents can still be similar if we only consider the angle. The smaller the angle is, the more similar the documents are. The cosine similarity and cosine distance have an inverse relationship. As the distance between two vectors increases, the similarity will decrease. Likewise, if the distance decreases, then the similarity between the two vectors increases. A·B is the product (dot) of the vectors A and B ||A|| and ||B|| is the length of the two vectors ||A|| * ||B|| is the cross product of the two vectors The cosine distance formula is then: 1 - Cosine Similarity
- #22: Distance Metrics in Vector Search https://weaviate.io/blog/distance-metrics-in-vector-search Cosine Similarity measures the similarity between documents regardless of the magnitude. This is advantageous because if two documents are far apart by the euclidean distance, the angle between them could still be small. For example, if the word ‘fruit' appears 30 times in one document and 10 in the other, that is a clear difference in magnitude, but the documents can still be similar if we only consider the angle. The smaller the angle is, the more similar the documents are. The cosine similarity and cosine distance have an inverse relationship. As the distance between two vectors increases, the similarity will decrease. Likewise, if the distance decreases, then the similarity between the two vectors increases.
- #23: Vector Databases simply explained! (Embeddings & Indexes) What is a Vector Database & How Does it Work? Use Cases + Examples | Pinecone Index in vector database Embedding is a list of numbers that represents data – a numerical representation of a text
- #24: What Is A Vector Database? | IBM Types of vector databases There are a few alternatives to choose from. Stand-alone, proprietary, fully vectorized databases such as Pinecone. Open-source solutions such as Weaviate or Milvus, which provide built-in RESTful APIs and support for Python and Java programming languages. Platforms with vector database capabilities integrated, such as IBM watsonx.data™. Vector database and database search extensions such as PostgreSQL’s open source pgvector extension, which provides vector similarity search capabilities. An SQL vector database can combine the advantages of a traditional SQL database with the power of a vector database.
- #25: What Is A Vector Database? | IBM Scalability store and manage massive amounts of unstructured data by scaling horizontally with additional nodes, maintaining performance as query demands and data volumes increase todo: Because they use high-dimensional vector embeddings, vector databases are better able to handle unstructured datasets.
- #27: Similarity search – find similar text, images, video, audio
- #28: What Is A Vector Database? | IBM „ Unstructured data—including social media posts, images, videos, audio clips and more—is growing 30% to 60% year over year.” Vector Databases vs. Relational Databases: Pros and Cons | Medium While relational databases can store data as JSON documents and offer flexibility, vector databases introduce specialized indexes for managing high-dimensional vectors, enabling them to excel in AI-driven applications.
- #31: About Chroma https://docs.trychroma.com Current version: 0.5.18
- #32: About Chroma https://docs.trychroma.com
- #33: Chroma - Vector Database for LLM Applications | OpenAI integration pip install chromadb - takes a few minutes (depending on Internet speed, because of a lot of downloading) - latest version: 0.5.18
- #35: Vector database Pinecone Embedding model multilinguage-e5-large
- #36: Vector database Pinecone Embedding model multilinguage-e5-large
- #37: Vector database Qdrant
- #38: Vector database Weaviate
- #39: Vector database Weaviate
- #41: They allow for search based on similarity