Bayesian classification
3 likes7,505 views
The document provides an overview of Bayesian classification and the Naive Bayes algorithm. It discusses how Thomas Bayes formulated Bayes' theorem and how the Naive Bayes classifier works by making the assumption that features are independent. The document explains the learning and testing phases of the Naive Bayes algorithm using examples, including how to handle continuous valued features. It also notes some issues like violating the independence assumption and how to prevent underflow problems in implementations.
Bayesian classification
- 1. Bayesian Classification Thomas Bayes (1701 – 7 April 1761) was an English statistician, philosopher and Presbyterian minister who is known for having formulated a specific case of the theorem that bears his name: Bayes' theorem. Bayes never published what would eventually become his most famous accomplishment, his notes were edited and published after his death by Richard Price Sir Thomas Bayes Slides by Manu Chandel, IIT Roorkee 1
- 2. Bayes Theorem Total Probability Bayes Theorem E1 E2 E3 …………………………… EN A 1. A is a outcome which can result from all the events E1, E2, ………… EN 2. All the events E1, E2, E3………. EN are mutually exclusive and exhaustive Slides by Manu Chandel, IIT Roorkee 2
- 3. Bayes Theorem Example Q. Given two bags each one having red and white balls. Both bags have equal chance of being chosen. If a ball is picked at random and found to be Red, what is the probability that the ball was chosen from bag A? Ans. Total probability of Red Ball = = = Probability that red ball was from Bag A ∗ ( ) = Slides by Manu Chandel, IIT Roorkee 3
- 4. Discriminative v/s Generative classifiers For a prediction function Discriminative classifiers estimate directly from the training data Generative classifiers estimate and directly from the training data. Naïve Bayes Classifier is a generative classifier Slides by Manu Chandel, IIT Roorkee 4
- 5. MAP Classification Rule Maximum A Posterior rule says that : “Jiski lathi uski bhains “ Input data belongs to the class whose is highest. Example : Suppose a news article is to be classified into following three categories: a) Politics b) Finance and c) Sports. So, X is our news article and three categories are denoted by Y1, Y2 and Y3 . Lets say , , then according to MAP classification rule news article will be classified into category 2 i.e. finance. Slides by Manu Chandel, IIT Roorkee 5
- 6. Naïve Bayes (Discrete values) An input to the classifier is often a feature vector containing various feature values e.g. A news article input to a news article classifier may be a vector of words. In Bayes classification we need to learn and from the given data. Here is feature vector with as feature values. Learning joint probability ( , ,…… , ) is difficult. Hence Naïve Bayes assumes that features are independent of each other. Assuming independence of features leads to ( , ,…… , ) Slides by Manu Chandel, IIT Roorkee 6
- 7. Naïve Bayes Algorithm (with Example) Learning phase of Naïve Bayes is represented by an example. Classifier needs to learn and for all Y Sr Year Height Pocket Money Grade Single 1 1 Average Low High Yes 2 2 Tall Average Low No 3 3 Short High High No 4 4 Average Average Low No 5 2 Tall High Low Yes 6 3 Tall Low High No 7 3 Average High Average Yes 8 1 Tall Average Average Yes 9 4 Short Average High Yes Data collected anonymously from BTECH Students IITR. Slides by Manu Chandel, IIT Roorkee 7
- 8. Naïve Bayes (Learning Phase ) Year ( = ) ( = ) 1 2/5 0 2 1/5 1/4 3 1/5 2/4 4 1/5 1/4 Height ( = ) ( = ) Tall 2/5 2/4 Short 1/5 1/4 Average 2/5 1/4 PM ( = ) ( = ) High 2/5 1/4 Low 1/5 1/4 Average 2/5 2/4 Grade ( = ) ( = ) High 2/5 2/4 Low 1/5 2/4 Average 2/5 0 Slides by Manu Chandel, IIT Roorkee 8
- 9. Naïve Bayes (Testing Phase) What will be the outcome of X= <4,Tall,Average,High> ? = = 4 ∗ = ∗ = ∗ = ∗ ( = ) = 1/5 * 2/5 * 2/5 * 2/5*5/9 = 0.00711 = = 4 ∗ = ∗ = ∗ = ∗ ( = ) = 1/4 * 2/4 * 2/4 * 2/4*4/9 = 0.0138 As 0.0138 > 0.00711 then X will be classified as Single = No Slides by Manu Chandel, IIT Roorkee 9
- 10. Naïve Bayes (Continuous Values ) Conditional probability often modeled with the normal distribution = = 1 2 exp(− ( − ) 2 ) = mean of feature values of = = standard deviation of feature values of = Learning Phase For = , , … … , = , , … . output Normal distributions. Test Phase Given an unknown instance = , , … . . , • Instead of looking-up tables, calculate conditional probabilities with all the normal distributions achieved in the learning phrase • Apply the MAP rule to make a decision Slides by Manu Chandel, IIT Roorkee 10
- 11. Naïve Bayes Continuous Value Example • Temperature is naturally of continuous value. • Yes: 25.2, 19.3, 18.5, 21.7, 20.1, 24.3, 22.8, 23.1, 19.8 • No: 27.3, 30.1, 17.4, 29.5, 15.1 • Estimate mean and variance for each class • and • • Learning phase output two Gaussian models for • . ( . ) . • . ( . ) . Slides by Manu Chandel, IIT Roorkee 11
- 12. Relevant Issues 1. Violation of independence Assumption 2. Zero Conditional Probability Problem If no example contains a feature value In this circumstances This can be solved by Slides by Manu Chandel, IIT Roorkee 12
- 13. Underflow Prevention • Multiplying lots of probabilities, which are between 0 and 1 by definition, can result in floating-point underflow. • Since it is better to perform all computations by summing logs of probabilities rather than multiplying probabilities. • Class with highest final un-normalized log probability score is still the most probable. Slides by Manu Chandel, IIT Roorkee 13
- 14. Summary • Naïve Bayes: the conditional independence assumption • Training is very easy and fast, just requiring considering each attribute in each class separately. • Test is straightforward, just looking up tables or calculating conditional probabilities with estimated distributions. • A popular generative model • Performance competitive to most of state-of-the-art classifiers even in presence of violating independence assumption. • Many successful applications, e.g., spam mail filtering Slides by Manu Chandel, IIT Roorkee 14