Nave Bias algorithm in Nature language processing

Editor's Notes

• #1: In this lecture we define the Naive Bayes classifier, a basic text classifier that will allow us to introduce many of the fundamental issues in text classification.
• #11: In this lecture we define the Naive Bayes classifier, a basic text classifier that will allow us to introduce many of the fundamental issues in text classification.
• #24: We've now seen the basic principles of naïve bayes classification!
• #34: In this lecture we'll do a worked example of naïve bayes sentiment analysis, and also introduce the binary multinominal naïve bayes algorithm.
• #62: In this lecture we introduce the concepts of precision and recall, and the F1 metric that combines them. These concepts are central to the evaluation of classifiers, and we'll use them throughout NLP, not just for Naïve Bayes but also for logistic regression and neural models.
• #63: Let's first consider binary classifiers. So that might include a simple binary decision about email (spam or not-spam). Or imagine that we are the proprietors of the Delicious Pie Company and we want to find out what people are saying about our pies on social media. We want to know if a particular social media post is talking about our pies (positive) or isn't (negative). To evaluate such a binary classifier, we'll need to know two things. What our classifier said about each email or post, and what it should have said, i.e. the correct answer, usually as defined by human labelers.
• #64: The first step is the confusion matrix a table for visualizing how an algorithm performs with respect to the human gold labels, We use two dimensions (system output and gold labels), and each cell labels a set of possible outcomes. In the pie detection case, for example, true positives are posts that are indeed about Delicious Pie (indicated by human-created gold labels) that our system correctly said were about pie. False negatives are posts that are indeed about pie but our system incorrectly labeled as pie. False positives are posts that aren't about pie but our system incorrectly said they were. And true negatives are non-pie-posts that are system correctly said were not about pie
• #65: . Here is the equation for accuracy: what percentage of all the observations (for the spam or pie examples that means all emails or tweets) our system labeled correctly. Although accuracy might seem a natural metric, we generally don’t use it for text classification tasks. Although accuracy might seem a natural metric, we generally don’t use it for text classification tasks.
• #66: Why don't we use accuracy? …Accuracy doesn't work well when we're dealing with uncommon or imbalanced classes. Suppose we look at a million social media posts to find Delicious Pie-lovers. Now most posts on the web are not about our pie, so let's imagine that only 100 posts are about our pie, and 999,900 are about something else. Now imagine we have the following really simple classifier: Every post is "not about pie"
• #67: Let's see what happens.
• #68: But this fabulous ‘no pie’ classifier would be completely useless, since it wouldn’t find a single one of the customer comments we are looking for. In other words, accuracy is not a good metric when the goal is to discover something that is rare, or at least not completely balanced in frequency, which is a very common situation in the world.
• #69: Precision is out of the things the system selected (the set of emails or tweets the sytstem claimed were positive, i.e. spam or pie-related), how many did it get right? how many were true positives, out of what I selected (true positives _ false positives). Recall is out of all the correct items that should have been positive, what % of them did the system select? So out of all the things that are gold positive, how many did the system find as true positives? So precision is about how much garbage we included in our findings; recall is more about making sure we didn't miss any treasure.
• #70: Recall and Precision will correctly evaluate our stupid "just say no" classifier as a bad classifier. The recall will be 0, since we returned no true positives out of the 100 true pie tweets (0 + 100). Precision is similarly 0 or in fact undefined, since both the numerator and denominator are 0. The metrics correctly assigns bad scores to our useless classifier. [note that this is a particular egregious example, but in normal situations there is typically a trade-off between precision and recall!] [to get high precision, a system should be very reluctant to guess – but then it may miss somethings and have poor recall] [to get high recall, a system shoul dbe very willing to guess– but then it may return some junk and have poor precision]
• #71: a combined measure: F-1: Why this function instead of just the arithmetic or geometric mean? F1 turns out to be the harmonic mean between precision and recall
• #72: F1 is a special case of the F-measure: weighted harmonic mean of precision and recall. The harmonic mean of a set of numbers is the reciprocal of the arithmetic mean of reciprocals. You can see here that F score is the harmonic mean, if we replace alpha with ½ we get 2/ (1/p + 1/r). The Harmonic mean of two values is closer to the minimum of the two numbers than arithmetic or geometric mean, so it weighs the lower of the two numbers more heavily. That is, if P and R are far apart, F will be nearer the lower value, which makes it a kind of conservative mean in this situation. Thus to do well on F1, you have to do well on BOTH P and R. Why the weights? in some applications you may care more about P or R. In practice we mainly use the balanced measure with beta = 1 and alpha =1/2
• #73: Lots of classification tasks have more than two classes, like sentiment could be 3-way. Consider the confusion matrix for a hypothetical 3-way email categorization decision (urgent, normal, spam). Notice that the system mistakenly labeled one spam document as urgent. We can compute distinct precision and recall values for each class. For example, the precision of the urgent category is 8 (the true positive urgents) over the true positives + false positives (the 10 normal and that 1 spam). The result, however, is 3 separate precision values and 3 separaterecall values!
• #74: To derive a single metric from these 3 sets of precisions and recalls, we can combine these in two ways.  In macroaveraging, we compute the performance for each class, and then average over classes. That is, we take the same precisions and recalls we computed on the previsions slide, one set for Urgent, one set for Normal, and one set for spam, and we just average those precisions to get a single macro average. In microaveraging, we instead first combine the decisions for all classes in a single confusion matrix, and then compute precision and recall from that table. Note that the microaverage is dominated by the more frequent class (in this case spam), since the counts are pooled and F1 is dominated by the count of true positives. The macroaverage better reflects the statistics of the smaller classes, and so is more appropriate when performance on all the classes is equally important. 
• #75: We'll be using precision, recall, and F1 constantly in evaluating NLP classifiers, since they are more appropriate than accuracy when classes are imbalanced.
• #76: For any of our NLP tasks, we'll want to think about how to avoid doing harm. Let's discuss some important considerations for classification.
• #78: One type of harm is called representational harm. These are harms that affect the representation of a group, for example demeaning them by perpetuating negative stereotypes about them. For example, Kititchenko and Mohammad in 2018 looked at representational harms in 200 sentiment analysis systems. They did this by creating pairs of sentences that were identicial except for a name. So a sentence might be "I talked to <person> yesterday". Then they created two versions of this sentence, one with a common African American name (like Shaniqua), and one with a common European American name (like Stephanie). Then they ran all the sentences in these sets of sentence pairs through 200 sentiment analysis systems. What they found is that systems assigned lower sentiment and more negative emotion to sentences with African American names This not only perpetuates negative stereotypes about African Americans, but means that a standard NLP tool like sentiment, widley used in marketing research, mental health studies, and so on, is likely to treats African Americans differently.
• #81: What causes these harms in text classification, and what can we do about it? These harms are partly caused by biases in the data; classifiers are trained on text that contains bias already. And research shows that NLP systems have a tendancy to amplify biases that already exist in the training data. The bias can come from the labels. Labels come from humans too, and humans have biases. And these harms can arise from the algorithms thsemlves These problems occur throughout NLP, even in the latest and most powerful neural models. Indeed, some research suggests that the bigger more powerful NLP models may exhibit even greater examples of these harms. There are no general-purpose mitigation algorithms or solutions. But harm mitigation is an active area of research throughout NLP.
• #82: Representational harms, censorship, and performance disparities, are just some of the harms that can be caused by text classification. It's important to examine our NLP tools for these harms and work to reduce them.