vectordb

A Vector Database (VectorDB) is designed to store and manage vector data, often used in machine learning and AI applications. Vector data refers to numerical representations of objects, which can be used for similarity search, clustering, and other tasks. A vector database is a collection of data that's stored as mathematical representations, or vector embeddings. Vector databases allow computers to

• Identify similarities: Compare data based on similarity metrics instead of exact matches

• Understand context: Identify relationships and draw comparisons

• Store and manipulate objects: Efficiently store and manipulate objects using vector embeddings

• Create indexes: Create indexes to facilitate fast searches

Vector databases are used to power: Search, Recommendations, Text generation, and Advanced artificial intelligence (AI) programs like large language models (LLMs).

Some examples of vector databases include: 

• Milvus: An open-source vector database that's optimized for handling high-dimensional data

• Weaviate: A vector search engine designed for natural language numerical data

• Elasticsearch: A Lucene-based distributed search engine that supports vector data

• Chroma: A vector database for building LLM apps

• Pinecone: A vector database

• Faiss: An open-source library for vector search created by Facebook

• Qdrant: A vector database

• pgvector: A vector database

How does a vector database work?

We all know how traditional databases work (more or less)—they store strings, numbers, and other types of scalar data in rows and columns. On the other hand, a vector database operates on vectors, so the way it’s optimized and queried is quite different.

In traditional databases, we are usually querying for rows in the database where the value usually exactly matches our query. In vector databases, we apply a similarity metric to find a vector that is the most similar to our query.

A vector database uses a combination of different algorithms that all participate in Approximate Nearest Neighbor (ANN) search. These algorithms optimize the search through hashing, quantization, or graph-based search.

These algorithms are assembled into a pipeline that provides fast and accurate retrieval of the neighbors of a queried vector. Since the vector database provides approximate results, the main trade-offs we consider are between accuracy and speed. The more accurate the result, the slower the query will be. However, a good system can provide ultra-fast search with near-perfect accuracy.