![COSINE DISTANCE
• Value: [-1.0, 1.0] (bigger is better)
• Slow
• Text document similarity
• Recommendation systems
a
b
𝜕](https://image.slidesharecdn.com/vector-search-250403081844-1760736c/85/Full-text-search-vector-search-or-both-16-320.jpg)
![HNSW IN
SQLITE
.load ./vec0
CREATE VIRTUAL TABLE document_embeddings USING vec0(
embedding FLOAT[768]
);
-- query
SELECT
rowid,
distance
FROM document_embeddings
WHERE embedding MATCH ‘[0.83443, 0.15224, …]’
ORDER BY distance](https://image.slidesharecdn.com/vector-search-250403081844-1760736c/85/Full-text-search-vector-search-or-both-28-320.jpg)
![HNSW IN
POSTGRESQL
CREATE EXTENSION vector;
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
embedding VECTOR(768)
);
-- create index
CREATE INDEX ON documents
USING hnsw(embedding vector_cosine_ops)
WITH (m=16, ef_construction=64);
-- query
SELECT
id,
embedding <=> ‘[0.83443, 0.15224, …]’ AS distance
FROM documents
WHERE distance > 0.7
ORDER BY distance DESC](https://image.slidesharecdn.com/vector-search-250403081844-1760736c/85/Full-text-search-vector-search-or-both-29-320.jpg)
Full text search, vector search or both?
- 1. FULL TEXT SEARCH, VECTOR SEARCH OR BOTH?
- 2. INTRODUCTION Bartosz Sypytkowski ▪ @horusiath@fosstodon.org ▪ b.sypytkowski@gmail.com ▪ bartoszsypytkowski.com
- 3. Full Text Search Vector Search Tips & Tricks Hybrid Search AGENDA
- 4. FULL TEXT INDEX lexically ordered list of “pages” containing the word “terms” not words “stop words” not included
- 5. PROBLEMS WITH FULL TEXT SEARCH 1. Stemming is language-specific – in .NET use NTextCat 2. Typographic errors – use n-gram similarity 3. Different words, same meaning 4. Text only
- 6. WHAT VECTOR SEARCH IS NOT
- 7. REFERENCES
- 9. VECTOR REPRESENTATION #1 AVERAGE COLOUR (RGB) 99 99 93 R G B Number of dimensions: 3 Dimension size: 8bit
- 10. VECTOR REPRESENTATION #2 AVERAGE COLOUR (CMYK) 0 0 6 61 C M Y Number of dimensions: 4 Dimension size: 0-100 K
- 11. VECTOR REPRESENTATION #3 COLOUR PALETTE A8B6BF 191A17 393836 637B97 7E6742 Number of dimensions: 5 Dimension size: 32bit
- 12. EMBEDDING MODELS Model Dimensions (float32) MTEB score OpenAI text-embedding-3-small 256 1536 62.0 62.3 OpenAI text-embedding-3-large 1024 3072 64.1 64.6 OpenAI text-embedding-ada-002 512 1536 61.6 61.0 Ollama all-minilm-l6-v2 384 56.26 Ollama mxbai-embed-large 1024 64.68 Gemini text-embedding-004 768 68.32
- 13. VECTOR SIMILARITY RIGHT TOOL FOR THE JOB
- 14. EUCLIDEAN DISTANCE • Values: [0, +Inf) (smaller is better) • Fast • Anomaly and fraud detection a b 𝑑(𝑎 ,𝑏)=√(𝑎 1−𝑏1)2+(𝑎2−𝑏2)2…+(𝑎𝑛−𝑏𝑛)2
- 15. DOT PRODUCT SIMILARITY • Value: (-Inf, +Inf) (bigger is better) • Fast • Image retrieval and matching • Music recommendation a b 𝜕 𝑎∙𝑏=|𝑎|∨ b∨ cos𝜕
- 16. COSINE DISTANCE • Value: [-1.0, 1.0] (bigger is better) • Slow • Text document similarity • Recommendation systems a b 𝜕
- 18. WHY DO WE NEED TO INDEX?
- 20. BUILDING HSNW INDEX Layer 0 JOIN EACH VECTOR WITH N CLOSEST NEIGHBOURS IN LAYER 0
- 21. BUILDING HSNW INDEX Layer 0 PROMOTE M% OF THE POINTS TO CONSTRUCT UPPER LAYER Layer 1
- 22. BUILDING HSNW INDEX Layer 0 Layer 1 JOIN EACH VECTOR WITH N CLOSEST NEIGHBOURS IN LAYER 1
- 24. USING HSNW INDEX Layer 0 Layer 1 query
- 25. USING HSNW INDEX Layer 0 Layer 1 query PICK THE CLOSEST NODE FROM TOP LAYER
- 26. USING HSNW INDEX Layer 0 Layer 1 query MOVE TO NEXT LAYER BELOW
- 27. USING HSNW INDEX Layer 0 Layer 1 query ONCE AT LAYER 0 PICK M RESULTS CLOSEST TO ORIGINAL QUERY
- 28. HNSW IN SQLITE .load ./vec0 CREATE VIRTUAL TABLE document_embeddings USING vec0( embedding FLOAT[768] ); -- query SELECT rowid, distance FROM document_embeddings WHERE embedding MATCH ‘[0.83443, 0.15224, …]’ ORDER BY distance
- 29. HNSW IN POSTGRESQL CREATE EXTENSION vector; CREATE TABLE documents ( id BIGSERIAL PRIMARY KEY, embedding VECTOR(768) ); -- create index CREATE INDEX ON documents USING hnsw(embedding vector_cosine_ops) WITH (m=16, ef_construction=64); -- query SELECT id, embedding <=> ‘[0.83443, 0.15224, …]’ AS distance FROM documents WHERE distance > 0.7 ORDER BY distance DESC
- 30. PROBLEM #1 EMBEDDING MODELS HAVE FIXED CONTEXT WINDOW
- 32. PROBLEM #2 HOW TO INDEX DOCUMENTS EDITED IN REAL TIME?
- 33. MULTI VECTOR INDEX 1. Cut content into paragraphs
- 34. MULTI VECTOR INDEX 1. Cut content into paragraphs 2. Group paragraphs into blocks by max allowed size (ie. 8000 chars).
- 35. MULTI VECTOR INDEX 1. Cut content into paragraphs 2. Group paragraphs into blocks by max allowed size (ie. 8000 chars). 3. Use content address hashing to identify blocks 0xae12e7 0x3902a1 0xef7312 0x06cd01
- 36. MULTI VECTOR INDEX 1. Cut content into paragraphs 2. Group paragraphs into blocks by max allowed size (ie. 8000 chars). 3. Use content address hashing to identify blocks 4. Reindex block only when sufficient change (i.e. >15%) was made. 0xae12e7 0x3902a1 0x43bf01 0x06cd01
- 37. MULTI VECTOR INDEX 1. Cut content into paragraphs 2. Group paragraphs into blocks by max allowed size (ie. 8000 chars). 3. Use content address hashing to identify blocks 4. Reindex block only when sufficient change (i.e. >15%) was made. 5. If block went under min allowed size (ie. 4000 chars), stich it to smallest adjacent block 0xae12e7 0x3902a1 0x9e721c
- 38. MULTI VECTOR INDEX 1. Cut content into paragraphs 2. Group paragraphs into blocks by max allowed size (ie. 8000 chars). 3. Use content address hashing to identify blocks 4. Reindex block only when sufficient change (i.e. >15%) was made. 5. If block went under min allowed size (ie. 4000 chars), stich it to smallest adjacent block 6. If block went over max allowed size, split it by paragraphs into two halves 0xae12e7 0x3902a1 0x12e757 0x49bcd1
- 39. PROBLEMS WITH VECTOR SEARCH 1. Weak at keyword search 2. Always produces results 3. CPU/Memory /Disk heavy 4. Complicated to maintain
- 40. HYBRID SEARCH RECIPROCAL RANK FUSION
- 41. RECIPROCAL RANK FUSION Query full text search vector search
- 42. RECIPROCAL RANK FUSION Query full text search vector search 1 A 2 B 3 C 4 D 5 E 1 C 2 B 3 F 4 A 5 D
- 43. RECIPROCAL RANK FUSION Query full text search vector search 1 A 2 B 3 C 4 D 5 E 1 C 2 B 3 F 4 A 5 D 1 1+𝑟𝑎𝑛𝑘𝐹𝑇𝑆 + 1 1+𝑟𝑎𝑛𝑘𝑉𝑆
- 44. RECIPROCAL RANK FUSION Query full text search vector search 1 A 2 B 3 C 4 D 5 E 1 C 2 B 3 F 4 A 5 D 1 1+𝑟𝑎𝑛𝑘𝐹𝑇𝑆 + 1 1+𝑟𝑎𝑛𝑘𝑉𝑆 0.75 C 0.70 A 0.66 B 0.37 D 0.25 F 0.17 E
- 45. SUMMARY
- 46. Azure AI Search: https://learn.microsoft.com/en-us/azure/search/vector-search-overview Hybrid search in Postgres: https://supabase.com/docs/guides/ai/hybrid-search Sqlite vector search extension: https://github.com/asg017/sqlite-vec HNSW Index explained: https://youtu.be/77QH0Y2PYKg REFERENCES
- 47. THANK YOU