Naive Bayes Classifier

Naïve Bayes Classifier
Arunabha Saha
Introduction
Classiﬁer Overview
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
Naive Bayes Classiﬁer
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
structure and functionalities
PGDM Business Analytics : Data Mining - I
Arunabha Saha1,2
1
Department of Computer Science & IT, Ananda Mohan College
2
Department of Computer Science, Vidyasagar College
July 29, 2019
Arunabha Saha (AMC, VC) Naïve Bayes Classifier July 29, 2019 1 / 15

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
what is classiﬁer
classiﬁcation is a supervised learning mechanism in which
the computer program learns from the given input dataset
and then use this experience to classify new observation.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
what is classiﬁer
it could be binary or multiclass classiﬁcation

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
what is classifier
it could be binary or multiclass classification
several types of classifiers :
– linear classifier: logistic regression, naïve bayes
classifer
– nearest neighbour
– support vector machine
– decision tree
– random forest
– neural networks

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
background
Naïve Bayes Classiﬁers(NBC) are a family of probabilistic
classiﬁcation algorithms based upon Bayes’ theorem with an
assumption of strong(naive) independence between the
features i.e. features are mutually exclusive or independent of
each other.
Bayes’ Theorem
Let A1, A2, ...., An are mutually exclusive events forming a
sample space S. Let B be any event from the same sample
space, such that P(B) > 0 then,
P(Ai |B) =
P(B|Ai )P(Ai )
n
j=1
P(B|Aj )P(Aj )

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Bayes’ Theorem
Simple form:
P(A|B) = P(B|A)P(A)
P(B)
where A and B are events and P(B) = 0
•P(A|B) is conditional probability; likelihood of event A
occurring given that B is true
•P(B|A) is conditional probability; likelihood of event B
occurring given that A is true
•P(A) and P(B) is the probability of event A and B;
independent of each other

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Bayes’ Theorem: Interpretation
Bayes’ theorem allows to update predicted probabilities of
an event by incorporating new information.1
Bayes’ theorem incorporates prior probability distributions
in order to generate posterior probabilities.
P(A) is the prior probability of event A, before new data is
collected.
P(A|B) is the posterior probability of event A occurring
given that event B has occurred.
1http://yudkowsky.net/rational/bayes

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Bayes’ Theorem: Example
An aircraft emergency locator transmitter (ELT) is a device
designed to transmit a signal in the case of a crash. The Altigauge
Manufacturing Co. makes 80% of the ELTs, the Bryant Co. makes
15% of them, and the Chartair Co. makes the other 5%. The ELTs
made by Altigauge have a 4% rate of defects, the Bryant ELTs have
a 6% rate of defects, and the Chartair ELTs have a 9% rate of
defects.
a. If an ELT is randomly selected from the general population of
all ELTs, ﬁnd the probability that it was made by the Altigauge
Manufacturing Co.
b. If a randomly selected ELT is then tested and is found to be
defective, ﬁnd the probability that it was made by the
Altigauge Manufacturing Co.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Solution: Bayes’ Rule
A = ELT manufactured by Altigauge
B = ELT manufactured by Bryant
C = ELT manufactured by Chartair
D = defected ELT
D = not defected ELT
a. ELT randomly selected, probability its from Altigauge P(A) = 0.8
b. ELT found defective, probability it is from Altigauge P(A|D);
prob. will be revised
P(A) = 0.80, P(B) = 0.15, P(C) = 0.05
P(D|A) = 0.04, P(D|B) = 0.06, P(D|C) = 0.09
P(A|D) = P(D|A)∗P(A)
[P(D|A)∗P(A)+P(D|B)∗P(B)+P(D|C)∗P(C)]
0.703
hence we have solved it using the Bayes’ rule directly.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Solution: Table method
Now we will solve it with explicit value.
Let total number if ELT manufactured is N = 10,000.
D D Total
A(Altigauge) 320 7680 8000
B(Bryant) 90 1410 1500
C(Chartair) 45 455 500
Total 455 9545 10,000
Now a ELT is found defective during the test and the probability of
it being from Altigauge is 320/455 0.703
hence the result in order with the result from the previous method.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Naïve Bayes Classiﬁer: Model
It is a conditional probability model. A problem instance to be
classiﬁed, n independent features (variables) represented by a vector
x = (x1, . . . , xn), it assigns to this instance probabilities,
P(Ck |x1, . . . , xn) for each of K possible outcomes.
Using Bayes’s theorem, the conditional probability can be expressed
as,
P(Ck |x) = P(Ck )P(x|Ck )
P(x)
it describes, posterior = (prior x likelihood)/evidence
The numerator is eqv. to joint probability P(Ck , x1, . . . , xn). Using
chain rule,
P(Ck , x1, . . . , xn) =
P(x1|x2, . . . , xn, Ck )P(x2|x3, . . . , xn, Ck ) . . . P(xn−1|xn, Ck )P(xn|Ck )P(Ck )

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Naïve Bayes Classifier: Model
naive assumption: are features in x are mutually independent
P(xi |xi+1, . . . , xn, Ck ) = P(xi |Ck ). So it expressed as,
P(Ck |x1, . . . , xn) ∝ P(Ck , x1, . . . , xn)
= P(Ck )
n
i=1
P(xi |Ck )
the conditional distribution over class variable C is,
P(Ck |x1, . . . , xn) = 1
Z
P(Ck )
n
i=1
P(xi |Ck )
where the evidence Z = P(x) =
k
P(Ck )P(x|Ck ) is constant if the
feature vector x is known.
constructing a classifier:
Hence the case is binary, but in case of multivariate classification we
need to find a class with maximum probability; it is known as
maximum a posteriori rule. Say assign a class label ˆy = Ck for
some k,
ˆy = argmax
k∈1,...,K
P(Ck )
n
i=1
P(xi |Ck )

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
NBC: Algorithm
Classiﬁcation Steps:
1 Read the dataset. It consists of two parts, feature matrix
and response vector.
2 Prepare the frequency table for each feature.
3 Prepare the likelihood table for each feature.
4 Apply Bayes’s rule to classify the response depending
upon the observation.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
NBC:Example

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
NBC:Example
now what is the probability of playing if
outcast = sunny
temp = hot
humidity = normal
wind = false

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
NBC:Example
now what is the probability of playing if
outcast = sunny
temp = hot
humidity = normal
wind = false
Limitations of NBC
incomplete training data : if any instance is 0 entire
conditional probability will collapse;
Laplace correction: asssign low probabilities so that prob.
computation doesn’t become 0
continuous variable: need to compute probability densities
non-independence of features: most important weakness
ignore correlated attributes; if its unknown which are
correlated then need to run correlation tests.

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
NBC: Application
Few applications are:
credit/loan approval analysis
medical diagnosis
weather forcasting
fraud analysis
spam ﬁltering

Arunabha Saha
Introduction
Background
Bayes’ Theorem
Interpretation
Example
Solution 1
Solution 2
NBC Model
NBC Algorithm
NBC example
NBC Appplication
End
Thank You

Naive Bayes Classifier

More Related Content

What's hot (20)

Similar to Naive Bayes Classifier (20)

More from Arunabha Saha (12)

Recently uploaded (20)

Naive Bayes Classifier