apidays Paris 2024 - Embeddings: Core Concepts for Developers, Jocelyn Matthews, Pinecone
0 likes88 views
Embeddings are numerical representations that capture essential features and relationships of objects in a continuous vector space, influenced by their context. They offer semantic meaning and efficient comparison of similarities, applicable in various domains like semantic search and anomaly detection. Key points include adaptability, dimensionality, and context sensitivity affecting their interpretation.
![Vectors
As developers, it might be easier to
think of a vector as an array
containing numerical values. For
example:
vector = [0,-2,...4]](https://image.slidesharecdn.com/videoparislondonlegalpresentationembeddingsjocelyn-241227150655-0ac17fc6/85/apidays-Paris-2024-Embeddings-Core-Concepts-for-Developers-Jocelyn-Matthews-Pinecone-7-320.jpg)
![Check the vectors
The distance between red and orange
is incredibly similar to blue and green…
But when we tested things trying to verify,
we got interesting results which show the
"understanding" of the relationship
This actually yields this
# Find a term that has the same distance and direction blue has from green, but starting from
blue
target_distance = distance_green_blue
target_direction = direction_green_blue
# Define a list of terms to compare
terms = ["red", "orange", "yellow", "green", "blue", "purple", "pink", "black", "white", "gray"]
# Get the embedding for each term
term_embeddings = {term: get_embedding(term) for term in terms}
# Find the term with the closest distance and same direction to the target distance and direction
closest_term = None
closest_distance = float('inf')
start_term = "red"
start_embedding = get_embedding(start_term)
for term, embedding in term_embeddings.items():
if term == start_term:
continue
distance, direction = cosine_distance_and_direction(start_embedding, embedding)
if direction == target_direction and abs(distance - target_distance) < closest_distance:
closest_distance = abs(distance - target_distance)
closest_term = term
closest_term, closest_distance](https://image.slidesharecdn.com/videoparislondonlegalpresentationembeddingsjocelyn-241227150655-0ac17fc6/85/apidays-Paris-2024-Embeddings-Core-Concepts-for-Developers-Jocelyn-Matthews-Pinecone-33-320.jpg)
![Check the vectors
The distance between red and orange
is incredibly similar to blue and green…
But when we played around to verify, we
got interesting results revealing the
semantic "understanding" of the
relationship
This actually yields this
# Find a term that has the same distance and direction blue
has from green, but starting from blue
target_distance = distance_green_blue
target_direction = direction_green_blue
# Define a list of terms to compare
terms = ["red", "orange", "yellow", "green", "blue",
"purple", "pink", "black", "white", "gray"]
# Get the embedding for each term
term_embeddings = {term: get_embedding(term) for term in
terms}
# Find the term with the closest distance and same direction
to the target distance and direction
closest_term = None
closest_distance = float('inf')
start_term = "red"
start_embedding = get_embedding(start_term)
for term, embedding in term_embeddings.items():
if term == start_term:
continue
distance, direction =
cosine_distance_and_direction(start_embedding, embedding)
if direction == target_direction and abs(distance -
target_distance) < closest_distance:
closest_distance = abs(distance - target_distance)
closest_term = term
closest_term, closest_distance
('purple', np.float64(0.006596347059928065))
Purple](https://image.slidesharecdn.com/videoparislondonlegalpresentationembeddingsjocelyn-241227150655-0ac17fc6/85/apidays-Paris-2024-Embeddings-Core-Concepts-for-Developers-Jocelyn-Matthews-Pinecone-34-320.jpg)
![Why not 'orange'?
The code's result of ('purple',
np.float64(0.006596347059928065))
suggests that, in the embedding space used by
the model, "red" and "purple" have a closer
semantic relationship than "red" and "orange".
The embedding model used in the code has
determined that "red" and "purple" are closer
semantically. This is likely due to the specific
contexts and relationships captured by the model
during training.
It yields 'purple' instead of orange because the
cosine distance and direction calculations
between the embeddings of "red" and other terms
result in "purple" being the closest match to the
target distance and direction from "green" to
"blue".
# Find a term that has the same distance and direction blue
has from green, but starting from blue
target_distance = distance_green_blue
target_direction = direction_green_blue
# Define a list of terms to compare
terms = ["red", "orange", "yellow", "green", "blue",
"purple", "pink", "black", "white", "gray"]
# Get the embedding for each term
term_embeddings = {term: get_embedding(term) for term in
terms}
# Find the term with the closest distance and same direction
to the target distance and direction
closest_term = None
closest_distance = float('inf')
start_term = "red"
start_embedding = get_embedding(start_term)
for term, embedding in term_embeddings.items():
if term == start_term:
continue
distance, direction =
cosine_distance_and_direction(start_embedding, embedding)
if direction == target_direction and abs(distance -
target_distance) < closest_distance:
closest_distance = abs(distance - target_distance)
closest_term = term
closest_term, closest_distance](https://image.slidesharecdn.com/videoparislondonlegalpresentationembeddingsjocelyn-241227150655-0ac17fc6/85/apidays-Paris-2024-Embeddings-Core-Concepts-for-Developers-Jocelyn-Matthews-Pinecone-35-320.jpg)
apidays Paris 2024 - Embeddings: Core Concepts for Developers, Jocelyn Matthews, Pinecone
- 2. Featured Speaker Jocelyn Matthews Head of Community, Pinecone Presented by jocelyn@pinecone.io 3
- 3. What are embeddings? Embeddings are numerical representations that capture the essential features and relationships of discrete objects, like words or documents, in a continuous vector space.
- 4. Embeddings: ● Are dynamic and context-sensitive. ● Capture the essence of the data they represent ● Are influenced by the context in which they are used ● Adaptability makes them powerful Humans think in sensations, words, ideas. Computers think in numbers
- 5. You don’t need to memorize this now Vector: a list of numbers that tell us about something Vector space: an environment in which vectors exist Semantics: the study of meaning communicated through language
- 6. Vectors A vector is a mathematical structure with a size and a direction. For example, we can think of the vector as a point in space, with the “direction” being an arrow from (0,0,0) to that point in the vector space.
- 7. Vectors As developers, it might be easier to think of a vector as an array containing numerical values. For example: vector = [0,-2,...4]
- 8. Vectors When we look at a bunch of vectors in one space, we can say that some are closer to one another, while others are far apart. Some vectors can seem to cluster together, while others could be sparsely distributed in the space.
- 9. An example you can bank on 🏦 Where is the Bank of England? 🌱 Where is the grassy bank? 🛩️ How does a plane bank? 🐝 “the bees decided to have a mutiny against their queen” 🐝 “flying stinging insects rebelled in opposition to the matriarch”
- 14. Word arithmetic king – man + woman = queen Image, Peter Sutor, “Metaconcepts: Isolating Context in Word Embeddings”
- 15. Word arithmetic king – man + woman = queen “Distributed Representations of Words and Phrases and their Compositionality”
- 16. Word arithmetic king – man + woman = queen “adding the vectors associated with the words king and woman while subtracting man is equal to the vector associated with queen. This describes a gender relationship.” – MIT Technology Review, 2015
- 17. Word arithmetic Paris - France + Poland = Warsaw
- 18. Word arithmetic Paris - France + Poland = Warsaw “In this case, the vector difference between Paris and France captures the concept of capital city.” – MIT Technology Review, 2015
- 19. Proximity
- 24. Green is to blue green blue
- 25. As orange is to… green orange blue
- 26. As orange is to…yep! green orange blue red
- 28. What’s The Fallacy? Why "Green : Blue :: Orange : Red" is Imperfect as a Teaching Tool • Simplicity of relationships • Linear vs nuanced • Lack of Context • How are the words used? • Dimensionality • 3D vs 100s of D • Oversimplification
- 29. What’s The Fallacy? Why "Green : Blue :: Orange : Red" is Imperfect as a Teaching Tool • Simplicity of relationships • Linear vs nuanced • Lack of Context • How are the words used? • Dimensionality • 3D vs 100s of D • Oversimplification
- 30. What’s The Fallacy? Why "Green : Blue :: Orange : Red" is Imperfect as a Teaching Tool • Simplicity of relationships • Linear vs nuanced • Lack of Context • How are the words used? • Dimensionality • 3D vs 100s of D • Oversimplification
- 31. What’s The Fallacy? Why "Green : Blue :: Orange : Red" is Imperfect as a Teaching Tool • Simplicity of relationships • Linear vs nuanced • Lack of Context • How are the words used? • Dimensionality • 3D vs 100s of D • Oversimplification
- 32. Life is Like a Box of… (Or, ”Check the Vectors”)
- 33. Check the vectors The distance between red and orange is incredibly similar to blue and green… But when we tested things trying to verify, we got interesting results which show the "understanding" of the relationship This actually yields this # Find a term that has the same distance and direction blue has from green, but starting from blue target_distance = distance_green_blue target_direction = direction_green_blue # Define a list of terms to compare terms = ["red", "orange", "yellow", "green", "blue", "purple", "pink", "black", "white", "gray"] # Get the embedding for each term term_embeddings = {term: get_embedding(term) for term in terms} # Find the term with the closest distance and same direction to the target distance and direction closest_term = None closest_distance = float('inf') start_term = "red" start_embedding = get_embedding(start_term) for term, embedding in term_embeddings.items(): if term == start_term: continue distance, direction = cosine_distance_and_direction(start_embedding, embedding) if direction == target_direction and abs(distance - target_distance) < closest_distance: closest_distance = abs(distance - target_distance) closest_term = term closest_term, closest_distance
- 34. Check the vectors The distance between red and orange is incredibly similar to blue and green… But when we played around to verify, we got interesting results revealing the semantic "understanding" of the relationship This actually yields this # Find a term that has the same distance and direction blue has from green, but starting from blue target_distance = distance_green_blue target_direction = direction_green_blue # Define a list of terms to compare terms = ["red", "orange", "yellow", "green", "blue", "purple", "pink", "black", "white", "gray"] # Get the embedding for each term term_embeddings = {term: get_embedding(term) for term in terms} # Find the term with the closest distance and same direction to the target distance and direction closest_term = None closest_distance = float('inf') start_term = "red" start_embedding = get_embedding(start_term) for term, embedding in term_embeddings.items(): if term == start_term: continue distance, direction = cosine_distance_and_direction(start_embedding, embedding) if direction == target_direction and abs(distance - target_distance) < closest_distance: closest_distance = abs(distance - target_distance) closest_term = term closest_term, closest_distance ('purple', np.float64(0.006596347059928065)) Purple
- 35. Why not 'orange'? The code's result of ('purple', np.float64(0.006596347059928065)) suggests that, in the embedding space used by the model, "red" and "purple" have a closer semantic relationship than "red" and "orange". The embedding model used in the code has determined that "red" and "purple" are closer semantically. This is likely due to the specific contexts and relationships captured by the model during training. It yields 'purple' instead of orange because the cosine distance and direction calculations between the embeddings of "red" and other terms result in "purple" being the closest match to the target distance and direction from "green" to "blue". # Find a term that has the same distance and direction blue has from green, but starting from blue target_distance = distance_green_blue target_direction = direction_green_blue # Define a list of terms to compare terms = ["red", "orange", "yellow", "green", "blue", "purple", "pink", "black", "white", "gray"] # Get the embedding for each term term_embeddings = {term: get_embedding(term) for term in terms} # Find the term with the closest distance and same direction to the target distance and direction closest_term = None closest_distance = float('inf') start_term = "red" start_embedding = get_embedding(start_term) for term, embedding in term_embeddings.items(): if term == start_term: continue distance, direction = cosine_distance_and_direction(start_embedding, embedding) if direction == target_direction and abs(distance - target_distance) < closest_distance: closest_distance = abs(distance - target_distance) closest_term = term closest_term, closest_distance
- 36. Embeddings TL;DR
- 37. What are embeddings? Embeddings are numerical representations that capture the essential features and relationships of discrete objects, like words or documents, in a continuous vector space.
- 38. The most important thing to understand Embeddings are numerical representations of data that: capture semantic meaning and allow for efficient comparison of similarity.
- 39. Key points about embeddings 1. They can represent various data types, not just text. 2. Dimensionality 3. Context sensitivity affects interpretation and application.
- 40. Applications of embeddings include: - Semantic search - Question-answering applications - Image search - Audio search - Recommender systems - Anomaly detection “Generate your own embeddings” (Inference API)
- 41. Sample app Legal Semantic Search
- 42. Sample app Shop the Look
- 43. © 2024 Pinecone – All rights reserved 45 1. Questions? #hallwaytrack 2. Recording? YouTube! 3. Slides? Ask me
- 46. © 2024 Pinecone – All rights reserved 48