Text similarity measures

Editor's Notes

• #2: Welcome to Week 3! Today we’ll be reviewing some key terms in NLP. 13
• #4: Today, we’ll be comparing how similar two pieces of text are. To do this, we’ll introduce the concepts of Levenshtein distance, document-term matrices and cosine similarity as ways of quantifying how ‘similar’ two pieces of text are. In addition, we’ll go over what the ‘Bag of Words’ assumption in NLP is and how Count Vectorizers and Term frequency-inverse document frequency help us create document-term matrices.
• #5: Before we start comparing two entire pieces of text, let’s start small, how do we even tell how ‘similar’ two individual words are?
• #6: Word similarity has a host of applications in natural language processing. <example?> Today, we’ll be covering the Levenshtein distance, the standard word similarity measure in practice.
• #7: This seems pretty easy at first. ‘MATH’ and ‘MATH’ are the same word, they’re 100% similar! ‘MATH’ and ‘BATH’ are only one letter apart, so they must be pretty similar too. However, as you go further down the list, it’s a bit trickier to quantify. There is actually a standard way of measuring distance though called Levenshtein distance.
• #8: As you can see, Levenshtein distance is pretty intuitive, there are only three operations and the computer finds the shortest way to edit one word into another. It’s similar to a childhood game - what’s the minimum number of steps you can take to change a word to another word?
• #9: Now, let’s reapply this to the example we showed earlier. Can you find the operations used to edit one word into the other with ‘Levenshtein distance’ number of steps?
• #10: Now, let’s go over TextBlob which is a toolkit which wraps around NLTK, making it easier to use. It has many of the same functions as NLTK, as long as a few new ones, which make it worth examining. You may also want to note that spaCy is an up and coming NLP library in Python It’s very powerful and like TextBlob, it’s easier to use than NLTK
• #11: As you can see, TextBlob is able to tokenize our text pretty easily!
• #12: As you can see, TextBlob is able to understand what we were trying to say! How does it do this? The Levenshtein distance we just introduced! Here are some more details on how TextBlob’s spell check works: http://norvig.com/spell-correct.html This is a very basic spell checker. It could definitely be smarter.
• #13: Now we move into the meat of the similarity measures section. Often, you’ll want to be finding the similarity between documents instead of just words.
• #14: Document similarity is used when you have a large amount of texts (a corpus) and you would like to perform some kind of analysis on them, without having to read them all in person. Document similarity measures allow you to encode the texts in a similar format and utilizing some cool maths, we can find a ‘distance’ between the texts to rate their similarities. To start though, we need to actually encode the documents. To do this, we’ll be leveraging the tokenization idea from last week and be introducing the idea of a ‘document-term’ matrix.
• #15: First let’s start with encoding a simple sentence (a short text). Let’s say we wanted to split up the sentence above. If we first extracted the words in the sentence (via tokenization) and we had a dictionary of all possible words, we could encode these tokens as vectors. This is known as the ‘one-hot encoding’ because as you can see, there is a ‘1’ in the vector where the word is present and a ‘0’ in every other vector not present in that record/document.
• #16: Now let’s move onto multiple texts! We’ll be using what’s known as a CountVectorizer to produce a document term matrix, which you can see at the bottom of the slide. Instantiation is easy as you can see, and its easy to turn the corpus that we made into a document-term matrix. Let’s analyze more closely. As you can see, our corpus has three texts, and our document-term matrix has three rows. In addition, our matrix has words along the top. Hopefully you can start to get an intuition for how our document-term matrix relates to our corpus from these insights.
• #17: So if you guessed that each ‘row’ corresponded to a document and each column corresponded to the terms in all the documents, you’d be right! What do the numbers mean? Well if we look at the top left entry, there’s a 0. That means that there are 0 occurences of the term ‘and’ in document 0. Doing a quick visual check, yep that seem’s right! Now, let’s do another example, look at the bottom left entry, it has a 1, meaning that document 2 (or text #3 because the matrix rows are 0-indexed) has exactly one occurrence of the word ‘and’. That also seem’s right! Quickly check the other entries if you aren’t convinced!
• #18: Now, this may seem like an overly simplistic encoding of a document to some of you. Why? Because all the information in our document seems to be able to be held in a single vector. The fact is that the vector is able to hold some of complexity of the text but it misses out on a lot. All that is encoded in a document term matrices is 1) which terms appear in a document and 2) how many times they appeared. If you had two documents [‘This is a text’, ‘Text this is a’], according to our encoding, they would be exactly the same! Basically, if you took the words in a text, put them in a bag and shook em around and looked at the output, to our encoding, it would be exactly the same. Thus, grammar and word order is ignored in the Bag of Words model. Although this seems extremely simplistic, we are actually able to perform some pretty powerful analysis using this assumption.
• #19: Let’s review what we’ve covered. To start, we tokenize the text into words, then used the count vectorizer to encode the texts into a document-term matrix. This means that we are using a ‘Bag of Words’ assumption because our encoding only cares about which words appear and how many times they appear.
• #20: How do we compare texts? We use cosine (yes the same formula from Linear Algebra). Why does that make sense? Remember, we encoded our text into vectors so to compare these vectors, we compute the cosine as a measure of how far our vectors are in our ‘concept space’. In our document-term matrix, the vector for document 1 would be the numbers in the row corresponding to document 1. This is true for every document, so we now have a vector for every document.
• #21: To make a cosine function, we just reproduce the formula using numpy functions.
• #22: Now, let’s try this on an example! Here are 5 texts that we want to compare. Let’s ask the question, which of these two texts seem the most similar? While many of the documents have the word “hot” in it, the two documents with “milk” actually seem most similar. Let’s see what our formula tells us.
• #23: First, we encode our corpus into a document term matrix . The reason we use the parameter stop_words=“english” is that our CountVectorizer has an automatic feature to remove stop_words in certain languages. Less work for us! Then, we convert this document-term matrix into a pandas DataFrame! Pandas is the premier data science library in python and if you aren’t familiar with it all, here’s a quick introduction to it included in the pandas documentation: https://pandas.pydata.org/pandas-docs/stable/10min.html
• #24: Here is our document-term matrix.
• #25: We use itertools to find all list of all the numerical combinations of size 2 in the number range of the length of the corpus using the ‘combinations’ function Next, we use those numerical combinations as indices and create a list of all the actual pairs of sentences. Next, we perform cosine similarity on all these pairs and create a list using that Finally, we sort them by their cosine similarity (in reverse so that the largest cosine similarity is presented first).
• #26: It seems that our cosine similarity chose these two documents as being the most similar, but looking at them, they don’t seem terribly similar. If we examine our document-term matrix by hand for these two documents, you can see that after removing stop words, these documents have very few differences. However, the word ‘milk’ seems to be a better differentiator of documents. I wonder if there’s a way to capture that?
• #27: As you just saw, the Count Vectorizer alone may be too simplistic for some purposes because every word is treated equally. A high frequency word that is not good for differentiation may be what’s dominating our similarity measure. Thus, we introduce the Term Frequency-Inverse Document Frequency metric to combat this.
• #28: Let’s break down this long and complicated name in the next two slides.
• #29: Term Frequency: How many times a term appears in a document. However, we don’t want to simply deal with raw counts, we want to weight the counts given how many terms are in the document in total. ----------------------------------------------------------- Want a sublinear transformation of the word count ?? <- what does this mean.
• #30: Remember, how we said we wanted a better differentiation metric? This is where the magic happens, we divide the total documents by the the number of documents containing the term we’re looking at. What happens as the number of documents containing the term grows towards the total number of documents? The rarer the word, the higher the IDF because it appears in less documents.
• #31: What does a TF-IDF score tell us about a word? What does it tell us about a document? TF-IDF assigns more weight to rare words and less weight to commonly occurring words. IDF is often written as log((D+1)/(d+1)) where: D = total documents (a document could be a line, paragraph, page etc. that represents a record/row in the dataset) d = documents contain the term
• #32: Notice that TF-IDF produces a score for each term/document combo. Thus, this is what we will use in place of the ‘count’ from before. So, our document vector will now be its TF-IDF score for every term. Notice that the score will still be 0 if the term doesn’t appear at all in the document. Our document similarity will now also be weighted towards rare words, and ‘similar’ documents will contain more ‘rare’ words in common.
• #33: Thankfully, not much is different from a code standpoint except for the Vectorizer that you import!
• #34: Our document-term matrix also looks very similar but instead of whole numbers as our counts, we now have TF-IDF scores which are floating point.
• #35: Remember, these were the two documents chosen as most similar by the CountVectorizer.
• #36: The code is same as before except for the TF-IDF vectorizer! Note that the values in the matrix are now TF-IDF values instead of counts
• #37: Now, let’s find the most similar documents! The two documents containing the word milk were rated as most similar because the our TF-IDF encoding allowed us to weight the rarer words, so we got the metric we wanted!
• #38: In summary, today we covered Levenshtein distance as a measure of word similarity and looked briefly at TextBlob. For document similarity we introduced the ideas of a document-term matrix, which can be produced either through a Count Vectorizer or TF-IDF Vectorizer, which weights rarer words. In addition, we looked at utilizing the cosine similarity between two document vectors to compare their similarity. Don’t forget that when we’re using a document-term matrix that we are under the ‘Bag of Words’ assumption. Our encoding does not care about word order or grammar!