5_LR_Apr_7_2021.pptx in nature language processing

Editor's Notes

• #1: In this lecture we talk about the difference between generative and discriminative classifiers, and the relationship between Naïve Bayes and Logistic Regression.
• #10: We've now seen the high-level intuition of the components of logistic regression and its relationship to the other classifier we've learned about, Naïve Bayes
• #11: In this lecture we'll see how to do classification with logistic regression and introduce the important sigmoid function
• #15: The weight wi represents how important that input feature is to the classification decision, and can be positive (providing evidence that the in- stance being classified belongs in the positive class) or negative (providing evidence that the instance being classified belongs in the negative class). Thus we might expect in a sentiment task the word awesome to have a high positive weight, and abysmal to have a very negative weight.
• #17: The bias term, also called the intercept, is another real number that’s added to the weighted inputs. In the rest of the book we’ll represent such sums using the dot product notation from linear algebra. The dot product of two vectors a and b, written as a · b is the sum of the products of the corresponding elements of each vector.
• #19: nothing in Eq. 5.3 forces z to be a legal probability, that is, to lie between 0 and 1. In fact, since weights are real-valued, the output might even be negative; z ranges from −∞ to ∞.
• #20: The sigmoid function The 1/1+e^-z (so-named because it looks like an s) is also called the logistic function. It takes a real value and maps it to the range [0, 1 It is nearly linear around 0 but outlier values get squashed toward 0 or 1.
• #22: We’re almost there. If we apply the sigmoid to the sum of the weighted features, we get a number between 0 and 1. To make it a probability, we just need to make sure that the two cases, p(y = 1) and p(y = 0), sum to 1. We can do this as follows:
• #23: The sigmoid function has the property
• #27: We've seen how logistic regression uses the sigmoid function to take weighted features for an input example x and assign it to the class 1 or 0.
• #28: Let's walk through an example using logistic regression to do sentiment classification
• #29: Suppose we are doing binary sentiment classification on movie review text, and we would like to know whether to assign the sentiment class 1=positive or 0=negative to the following review
• #30: We’ll represent each input observation by the 6 features x1...x6 of the input
• #31: Let’s assume for the moment that we’ve already learned a real-valued weight for each of these features, and that the 6 weights corresponding to the 6 features are as follows. The weight w1, for example indicates how important a feature the number of positive lexicon words (great, nice, enjoyable, etc.) is to a positive sentiment decision, while w2 tells us the importance of negative lexicon words. Note that w1 = 2.5 is positive, while w2 = −5.0, meaning that negative words are negatively associated with a positive sentiment decision, and are about twice as important as positive words.
• #32: Given these 6 features and the input review x, we can compute P(+|x) and P(−|x)
• #33: We might use features like x1 below expressing that the current word is lower case and the class is EOS (perhaps with a positive weight), or that the current word is in our abbreviations dictionary (“Prof.”) and the class is EOS (perhaps with a negative weight). A feature can also express a quite complex combination of properties. For example a period following an upper case word is likely to be an EOS, but if the word itself is St. and the previous word is capitalized, then the period is likely part of a shortening of the word street.
• #35: We've now seen the details of how logistic regression can take feature values and weights to assign a class to an input.
• #36: Let's now turn to learning the parameters for logistic regression. We'll start with the cross-entropy loss function
• #37: Logistic regression is an instance of supervised classification in which we know the correct label y (either 0 or 1) for each observation x. But what the system produces is an estimate, ^y. We want to learn parameters (meaning w and b) that make yˆ for each training observation as close as possible to the true y.
• #38: This requires two components. The first is a metric for how close the current label (yˆ) is to the true gold label y. Rather than measure similarity, we usually talk about the opposite of this: the distance between the system output and the gold output, and we call this distance the loss function or the cost function. We'll we’ll introduce the loss function that is commonly used for logistic regression and also for neural networks, the cross-entropy loss. The second thing we need is an optimization algorithm for iteratively updating the weights so as to minimize this loss function. The standard algorithm for this is gradient descent; we’ll introduce the stochastic gradient descent algorithm in the following section.
• #39: We need a loss function that expresses, for an observation x, how close the classifier output (yˆ = σ (w · x + b)) is to the correct output (y, which is 0 or 1).
• #40: We do this via a loss function that prefers the correct class labels of the training examples to be more likely. This is called conditional maximum likelihood estimation: we choose the parameters w,b that maximize the log probability of the true y labels in the training data given the observations x. The resulting loss function is the negative log likelihood loss, generally called the cross-entropy loss.
• #41: Let’s derive this loss function, applied to a single observation x. We’d like to learn weights that maximize the probability of the correct label, that's p(y|x).
• #46: By contrast, let’s pretend instead that the example was actually negative, i.e., the true y = 0 (perhaps the reviewer went on to say “But bottom line, the movie is terrible! I beg you not to see it!”) . In this case our model is confused, the model is wrong, so we’d want the loss to be higher. Let's plug in y = 0 and 1−σ(w·x+b) = .31
• #48: We've derived the cross-entropy loss and seen how it applies in our sentiment example. Cross entropy loss is equally important for neural networks.
• #49: In this lecture we introduce the stochastic gradient descent algorithm used for optimizing the weights for logistic regression and neural networks.
• #50: Our goal with gradient descent is to find the optimal weights: minimize the loss function we’ve defined for the model. We’ll explicitly represent the fact that the loss function L is parameterized by the weights, which we can refer to in machine learning in general as θ (in the case of logistic regression θ = w, b). And we’ll represent as f (x; θ ) to make the dependence on θ more obvious. So the goal is to find the set of weights which minimizes the loss function, averaged over all examples: EQ
• #51: How do we find the minimum of a loss function? Gradient descent is a method that finds a minimum of a function by figuring out in which direction (in the space of the parameters θ) the function’s slope is rising the most steeply, and moving in the opposite direction. The intuition is that if you are hiking in a canyon and trying to descend most quickly down to the river at the bottom, you might look around yourself 360 degrees, find the direction where the ground is sloping the steepest, and walk downhill in that direction.
• #52: For logistic regression, this loss function is conveniently convex. A convex function has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum. (By contrast, the loss for multi-layer neural networks is non-convex, and gradient descent may get stuck in local minima for neural network training and never find the global optimum.)
• #53: Although the algorithm (and the concept of gradient) are designed for direction vectors, let’s first consider a visualization of the case where the parameter of our system is just a single scalar w. Given a random initialization of w at some value w1, and assuming the loss function L happened to have this shape, we need the algorithm to tell us whether at the next iteration we should move left (making w2 smaller than w1) or right (making w2 bigger than w1) to reach the minimum. The gradient descent algorithm answers this question by finding the gradient of the loss function at the current point and moving in the opposite direction.
• #54: The gradient of a function of many variables is a vector pointing in the direction of the greatest increase in a function. The gradient is a multi-variable generalization of the slope, so for a function of one variable like the one in this figure, we can informally think of the gradient as the slope. The dotted line shows the slope of this hypothetical loss function at point w = w1 . You can see that the slope of this dotted line is negative.
• #55: Thus to find the minimum, gradient descent tells us to go in the opposite direction: moving w in a positive direction.
• #57: The magnitude of the amount to move in gradient descent is the value of the slope of the loss function (with respect to w) weighted by a learning rate η . A higher (faster) learning rate means that we should move w more on each step. The change we make in our parameter is the learning rate times the gradient (or the slope, in our single-variable example)
• #58: Now let’s extend the intuition from a function of one scalar variable w to many variables, because we don’t just want to move left or right, we want to know where in the N-dimensional space (of the N parameters that make up θ ) we should move. The gradient is just such a vector; it expresses the directional components of the sharpest slope along each of those N dimensions.
• #59: If we’re just imagining two weight dimensions (say for one weight w and one bias b), the gradient might be a vector with two orthogonal components, each of which tells us how much the ground slopes in the w dimension and in the b dimension. This figure shows a visualization of the value of a 2-dimensional gradient vector taken at the red point.
• #60: In an actual logistic regression, the parameter vector w is much longer than 1 or 2, since the input feature vector x can be quite long, and we need a weight wi for each xi. For each dimension/variable wi in w (plus the bias b), the gradient will have a component that tells us the slope with respect to that variable. Essentially we’re asking: “How much would a small change in that variable wi influence the total loss function L?” In each dimension wi, we express the slope as a partial derivative of the loss function with respect to wi. The gradient is then defined as a vector of these partials.
• #61: Here's a vector of gradients. We’ll represent yˆ as f (x; θ ) to make the dependence on θ more obvious:
• #62: In order to update θ , we need a definition for the gradient ∇L( f (x; θ ), y). Recall that for logistic regression, the cross-entropy loss function is: EQ It turns out that the derivative of this function for one observation vector x is EQ Note that the gradient with respect to a single weight w j represents a very intuitive value: the difference between the true y and our estimated yˆ = σ (w · x + b) for that observation, multiplied by the corresponding input value x j .
• #63: Stochastic gradient descent is an online algorithm that minimizes the loss function by computing its gradient after each training example, and nudging θ in the right direction (the opposite direction of the gradient). The algorithm can terminate when it converges (or when the gradient norm < ε), or when progress halts (for example when the loss starts going up on a held-out set).
• #64: The learning rate η is a hyperparameter that must be adjusted. If it’s too high, the learner will take steps that are too large, overshooting the minimum of the loss function. If it’s too low, the learner will take steps that are too small, and take too long to get to the minimum. It is common to start with a higher learning rate and then slowly decrease it. Hyperparameters are a special kind of parameter for any machine learning model. Unlike regular parameters of a model (weights like w and b), which are learned by the algorithm from the training set, hyperparameters are special parameters chosen by the algorithm designer that affect how the algorithm works.
• #65: We've now introduced the important stochastic gradient descent algorithm. We'll give some more details in the next lecture.
• #66: In this lecture we'll walk through an example of stochastic descent and give a few more details.
• #67: Let’s walk though a single step of the gradient descent algorithm. We’ll use a simplified version of our sentiment classification example as it sees a single observation x, whose correct value is y = 1 (this is a positive review), and with only two features: x1  = 3 (count of positive lexicon words), and x2  = 2 (count of negative lexicon words). Let’s assume the initial weights and bias in θ_0 are all set to 0, and the initial learning rate η is 0.1:
• #68: Here's our update equation for SGD. In order to do this update to theta, we'll need to know the gradient of the loss function. In our mini example there are three parameters, so the gradient vector has 3 dimensions, for w1, w2, and b. We can compute the first gradient as follows:
• #76: Note that this observation x happened to be a positive example. We would expect that after seeing more negative examples with high counts of negative words, that the weight w2, the weight for the "negative lexicon feature', would shift to have a negative value.
• #77: Stochastic gradient descent is called stochastic because it chooses a single random example at a time, moving the weights so as to improve performance on that single example. That can result in very choppy movements, so it’s common to compute the gradient over batches of training instances rather than a single instance. For example in batch training we compute the gradient over the entire dataset. By seeing so many examples, batch training offers a superb estimate of which direction to move the weights, at the cost of spending a lot of time processing every single example in the training set to compute this perfect direction. A compromise is mini-batch training: we train on a group of m examples (perhaps 512, or 1024) that is less than the whole dataset. Mini-batch training also has the advantage of computational efficiency. The mini-batches can easily be vectorized, choosing the size of the mini-batch based on the computational resources. This allows us to process all the exam- ples in one mini-batch in parallel and then accumulate the loss, something that’s not possible with individual or batch training.
• #78: We've now seen the stochastic gradient descent algorithm and discussed variants like mini-batch training.