Bayes Theorem.pdf
- 1. Bayesian Learning Each observed training example can incrementally decrease or increase the estimated probability that a hypothesis is correct. This provides a more flexible approach to learning than algorithms that completely eliminate a hypothesis if it is found to be inconsistent with any single example. Prior knowledge can be combined with observed data to determine the final probability of a hypothesis. In Bayesian learning, prior knowledge is provided by asserting A prior probability for each candidate hypothesis, and a probability distribution over observed data for each possible hypothesis. Bayesian methods can accommodate hypotheses that make probabilistic predictions New instances can be classified by combining the predictions of multiple hypotheses, weighted by their probabilities. Even in cases where Bayesian methods prove computationally intractable, they can provide a standard of optimal decision making against which other practical methods can be measured
- 2. Bayesian Learning… Require initial knowledge of many probabilities – When these probabilities are not known in advance, they are often estimated based on background knowledge, previously available data, and assumptions about the form of the underlying distributions. Significant computational cost is required to determine the Bayes optimal hypothesis in the general case (linear in the number of candidate hypotheses). In certain specialized situations, this computational cost can be significantly reduced
- 3. Bayes Theorem In machine learning, we try to determine the best hypothesis from some hypothesis space H, given the observed training data D. In Bayesian learning, the best hypothesis means the most probable hypothesis, given the data D plus any initial knowledge about the prior probabilities of the various hypotheses in H. Bayes theorem provides a way to calculate the probability of a hypothesis based on its prior probability, the probabilities of observing various data given the hypothesis, and the observed data itself
- 4. Bayes Theorem.. In ML problems, we are interested in the probability P(h|D) that h holds given the observed training data D. Bayes theorem provides a way to calculate the posterior probability P(h|D), from the prior probability P(h), together with P(D) and P(D|h). Bayes Theorem: P(h) is prior probability of hypothesis h P(D) is prior probability of training data D P(h|D) is posterior probability of h given D P(D|h) is posterior probability of D given h P(h|D) increases with P(h) and P(D|h) according to Bayes theorem. P(h|D) decreases as P(D) increases, because the more probable it is that D will be observed independent of h, the less evidence D provides in support of h
- 5. Maximum A Posteriori (MAP) Hypothesis, hMAP
- 11. Maximum Likelihood and Least- Squared Error Hypotheses Many learning approaches such as neural network learning, linear regression, and polynomial curve fitting try to learn a continuous-valued target function. Under certain assumptions any learning algorithm that minimizes the squared error between the output hypothesis predictions and the training data will output a MAXIMUM LIKELIHOOD HYPOTHESIS. The significance of this result is that it provides a Bayesian justification (under certain assumptions) for many neural network and other curve fitting methods that attempt to minimize the sum of squared errors over the training data.
- 12. Maximum Likelihood and Least- Squared Error Hypotheses – Deriving hML
- 13. Maximum Likelihood and Least- Squared Error Hypotheses – Deriving hML
- 14. Maximum Likelihood and Least- Squared Error Hypotheses – Deriving hML
- 15. Maximum Likelihood and Least- Squared Error Hypotheses The maximum likelihood hypothesis hML is the one that minimizes the sum of the squared errors between observed training values di and hypothesis predictions h(xi ). This holds under the assumption that the observed training values di are generated by adding random noise to the true target value, where this random noise is drawn independently for each example from a Normal distribution with zero mean. Similar derivations can be performed starting with other assumed noise distributions, producing different results. Why is it reasonable to choose the Normal distribution to characterize noise? – One reason, is that it allows for a mathematically straightforward analysis. – A second reason is that the smooth, bell-shaped distribution is a good approximation to many types of noise in physical systems. Minimizing the sum of squared errors is a common approach in many neural network, curve fitting, and other approaches to approximating real-valued functions.
- 17. Minimum Description Length Principle Consider the problem of designing a code to transmit messages drawn at random, where the probability of encountering message i is pi. We are interested here in the most compact code; that is, we are interested in the code that minimizes the expected number of bits we must transmit in order to encode a message drawn at random. Clearly, to minimize the expected code length we should assign shorter codes to messages that are more probable. We will refer to the number of bits required to encode message i using code C as the description length of message i with respect to C, which we denote by Lc(i).
- 18. Minimum Description Length Principle • Therefore the above equation is rewritten to show that is the hypothesis h that minimizes the sum given by the description length of the hypothesis plus the description length of the data given the hypothesis
- 19. Bayes Optimal Classifier It is the most probable classification for the new instance in the given the training data. So, whenever it gets the new instance, it makes the most efficient classification. To develop some intuitions, consider a hypothesis space containing three hypotheses, h1, h2, and h3. Suppose that the posterior probabilities of these hypotheses given the training data are .4, .3, and .3 respectively. Thus, h1 is the MAP hypothesis. Suppose a new instance x is encountered, which is classified positive by h1, but negative by h2 and h3. Taking all hypotheses into account, the probability that x is positive is .4 (the probability associated with hi), and the probability that it is negative is therefore .6. The most probable classification (negative) in this case is different from the classification generated by the MAP hypothesis If the possible classification of the new example can take on any value vj from some set V, then the probability P(vjlD) that the correct classification for the new instance is vj is,
- 21. Gibbs Algorithm