Bayesian Networks Model in Step By Steps

Weng-Keen Wong, Oregon State University ©2005
1
Bayesian Networks: A Tutorial
Weng-Keen Wong
School of Electrical Engineering and Computer Science
Oregon State University
Modified by
Longin Jan Latecki
Temple University
latecki@temple.edu

2
Introduction
Suppose you are trying to determine
if a patient has inhalational
anthrax. You observe the
following symptoms:
• The patient has a cough
• The patient has a fever
• The patient has difficulty
breathing

3
Introduction
You would like to determine how
likely the patient is infected with
inhalational anthrax given that the
patient has a cough, a fever, and
difficulty breathing
We are not 100% certain that the
patient has anthrax because of these
symptoms. We are dealing with
uncertainty!

4
Introduction
Now suppose you order an x-ray
and observe that the patient has a
wide mediastinum.
Your belief that that the patient is
infected with inhalational anthrax is
now much higher.

5
Introduction
• In the previous slides, what you observed
affected your belief that the patient is
infected with anthrax
• This is called reasoning with uncertainty
• Wouldn’t it be nice if we had some
methodology for reasoning with
uncertainty? Well in fact, we do…

6
Bayesian Networks
• In the opinion of many AI researchers, Bayesian
networks are the most significant contribution in
AI in the last 10 years
• They are used in many applications eg. spam
filtering, speech recognition, robotics, diagnostic
systems and even syndromic surveillance
HasAnthrax
HasCough HasFever HasDifficultyBreathing HasWideMediastinum

7
Outline
1. Introduction
2. Probability Primer
3. Bayesian networks

8
Probability Primer: Random Variables
• A random variable is the basic element of
probability
• Refers to an event and there is some degree
of uncertainty as to the outcome of the
event
• For example, the random variable A could
be the event of getting a head on a coin flip

9
Boolean Random Variables
• We will start with the simplest type of random
variables – Boolean ones
• Take the values true or false
• Think of the event as occurring or not occurring
• Examples (Let A be a Boolean random variable):
A = Getting a head on a coin flip
A = It will rain today

10
The Joint Probability Distribution
• Joint probabilities can be between
any number of variables
eg. P(A = true, B = true, C = true)
• For each combination of variables,
we need to say how probable that
combination is
• The probabilities of these
combinations need to sum to 1
A B C P(A,B,C)
false false false 0.1
false false true 0.2
false true false 0.05
false true true 0.05
true false false 0.3
true false true 0.1
true true false 0.05
true true true 0.15
Sums to 1

11
• Once you have the joint probability
distribution, you can calculate any
probability involving A, B, and C
• Note: May need to use
marginalization and Bayes rule,
(both of which are not discussed in
these slides)
A B C P(A,B,C)
true false true 0.1
true true true 0.15
Examples of things you can compute:
• P(A=true) = sum of P(A,B,C) in rows with A=true
• P(A=true, B = true | C=true) =
P(A = true, B = true, C = true) / P(C = true)

12
The Problem with the Joint
Distribution
• Lots of entries in the
table to fill up!
• For k Boolean random
variables, you need a
table of size 2k
• How do we use fewer
numbers? Need the
concept of
independence
A B C P(A,B,C)
true false true 0.1
true true true 0.15

13
Independence
Variables A and B are independent if any of
the following hold:
• P(A,B) = P(A) P(B)
• P(A | B) = P(A)
• P(B | A) = P(B)
This says that knowing the outcome of
A does not tell me anything new about
the outcome of B.

14
Independence
How is independence useful?
• Suppose you have n coin flips and you want to
calculate the joint distribution P(C1, …, Cn)
• If the coin flips are not independent, you need 2n
values in the table
• If the coin flips are independent, then



n
i
i
n C
P
C
C
P
1
1 )
(
)
,...,
( Each P(Ci) table has 2 entries
and there are n of them for a
total of 2n values

15
Conditional Independence
Variables A and B are conditionally
independent given C if any of the following
hold:
• P(A, B | C) = P(A | C) P(B | C)
• P(A | B, C) = P(A | C)
• P(B | A, C) = P(B | C)
Knowing C tells me everything about B. I don’t gain
anything by knowing A (either because A doesn’t
influence B or because knowing C provides all the
information knowing A would give)

16
Outline
1. Introduction
2. Probability Primer
3. Bayesian networks

A Bayesian Network
A Bayesian network is made up of:
A P(A)
false 0.6
true 0.4
A
B
C D
A B P(B|A)
false false 0.01
false true 0.99
true false 0.7
true true 0.3
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1
B D P(D|B)
false false 0.02
false true 0.98
true false 0.05
true true 0.95
1. A Directed Acyclic Graph
2. A set of tables for each node in the graph

18
A Directed Acyclic Graph
A
B
C D
Each node in the graph is a
random variable
A node X is a parent of
another node Y if there is an
arrow from node X to node Y
eg. A is a parent of B
Informally, an arrow from
node X to node Y means X
has a direct influence on Y

A Set of Tables for Each Node
Each node Xi has a
conditional probability
distribution P(Xi | Parents(Xi))
that quantifies the effect of
the parents on the node
The parameters are the
probabilities in these
conditional probability tables
(CPTs)
A P(A)
false 0.6
true 0.4
A B P(B|A)
false false 0.01
false true 0.99
true false 0.7
true true 0.3
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1
B D P(D|B)
false false 0.02
false true 0.98
true false 0.05
true true 0.95
A
B
C D

20
A Set of Tables for Each Node
Conditional Probability
Distribution for C given B
If you have a Boolean variable with k Boolean parents, this table
has 2k+1 probabilities (but only 2k need to be stored)
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1 For a given combination of values of the parents (B
in this example), the entries for P(C=true | B) and
P(C=false | B) must add up to 1
eg. P(C=true | B=false) + P(C=false |B=false )=1

21
Bayesian Networks
Two important properties:
1. Encodes the conditional independence
relationships between the variables in the
graph structure
2. Is a compact representation of the joint
probability distribution over the variables

22
Conditional Independence
The Markov condition: given its parents (P1, P2),
a node (X) is conditionally independent of its non-
descendants (ND1, ND2)
X
P1 P2
C1 C2
ND2
ND1

23
Due to the Markov condition, we can compute
the joint probability distribution over all the
variables X1, …, Xn in the Bayesian net using
the formula:






n
i
i
i
i
n
n X
Parents
x
X
P
x
X
x
X
P
1
1
1 ))
(
|
(
)
,...,
(
Where Parents(Xi) means the values of the Parents of the node Xi
with respect to the graph

24
Using a Bayesian Network Example
Using the network in the example, suppose you want to
calculate:
P(A = true, B = true, C = true, D = true)
= P(A = true) * P(B = true | A = true) *
P(C = true | B = true) P( D = true | B = true)
= (0.4)*(0.3)*(0.1)*(0.95) A
B
C D

25
Using a Bayesian Network Example
Using the network in the example, suppose you want to
calculate:
= (0.4)*(0.3)*(0.1)*(0.95) A
B
C D
This is from the
graph structure
These numbers are from the
conditional probability tables

26
Joint Probability Factorization
For any joint distribution of random variables the following
factorization is always true:
We derive it by repeatedly applying the Bayes’ Rule
P(X,Y)=P(X|Y)P(Y):
)
,
,
|
(
)
,
|
(
)
|
(
)
(
)
(
)
|
(
)
,
|
(
)
,
,
|
(
)
(
)
|
(
)
,
|
,
(
)
(
)
|
,
,
(
)
,
,
,
(
C
B
A
D
P
B
A
C
P
A
B
P
A
P
A
P
A
B
P
A
B
C
P
A
B
C
D
P
A
P
A
B
P
A
B
D
C
P
A
P
A
D
C
B
P
D
C
B
A
P



)
,
,
|
(
)
,
|
(
)
|
(
)
(
)
,
,
,
( C
B
A
D
P
B
A
C
P
A
B
P
A
P
D
C
B
A
P 

27
Joint Probability Factorization
A
B
C D
)
|
(
)
|
(
)
|
(
)
(
)
,
,
|
(
)
,
|
(
)
|
(
)
(
)
,
,
,
(
B
D
P
B
C
P
A
B
P
A
P
C
B
A
D
P
B
A
C
P
A
B
P
A
P
D
C
B
A
P


Our example graph carries additional independence
information, which simplifies the joint distribution:
This is why, we only need the tables for
P(A), P(B|A), P(C|B), and P(D|B)
and why we computed
= (0.4)*(0.3)*(0.1)*(0.95)

28
Inference
• Using a Bayesian network to compute
probabilities is called inference
• In general, inference involves queries of the form:
P( X | E )
X = The query variable(s)
E = The evidence variable(s)

29
Inference
• An example of a query would be:
P( HasAnthrax = true | HasFever = true, HasCough = true)
• Note: Even though HasDifficultyBreathing and
HasWideMediastinum are in the Bayesian network, they are
not given values in the query (ie. they do not appear either as
query variables or evidence variables)
• They are treated as unobserved variables and summed out.
HasAnthrax
HasCough HasFever HasDifficultyBreathing HasWideMediastinum

Inference Example
A P(A)
false 0.6
true 0.4
A
B
C D
A B P(B|A)
false false 0.01
false true 0.99
true false 0.7
true true 0.3
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1
B D P(D|B)
false false 0.02
false true 0.98
true false 0.05
true true 0.95
)
(
)
,
,
,
(
)
(
)
,
(
)
|
( ,
t
A
P
d
D
t
C
b
B
t
A
P
t
A
P
t
C
t
A
P
t
A
t
C
P d
b













Supposed we know that A=true.
What is more probable C=true or D=true?
For this we need to compute
P(C=t | A =t) and P(D=t | A =t).
Let us compute the first one.

What is P(A=true)?
A P(A)
false 0.6
true 0.4
A
B
C D
A B P(B|A)
false false 0.01
false true 0.99
true false 0.7
true true 0.3
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1
B D P(D|B)
false false 0.02
false true 0.98
true false 0.05
true true 0.95
...
))
|
(
)
|
(
)
|
(
)
|
(
(
4
.
0
1
*
)
|
(
)
|
(
)
(
)
|
(
)
|
(
)
|
(
)
(
)
|
(
)
|
(
)
|
(
)
(
)
|
(
)
|
(
)
|
(
)
(
)
|
(
)
|
(
)
|
(
)
(
)
,
,
,
(
)
(
,
,
,
,
,
,
,



































































f
B
c
C
P
t
A
f
B
P
t
B
c
C
P
t
A
t
B
P
b
B
c
C
P
t
A
b
B
P
t
A
P
b
B
d
D
P
b
B
c
C
P
t
A
b
B
P
t
A
P
b
B
d
D
P
b
B
c
C
P
t
A
b
B
P
t
A
P
b
B
d
D
P
b
B
c
C
P
t
A
b
B
P
t
A
P
b
B
d
D
P
b
B
c
C
P
t
A
b
B
P
t
A
P
d
D
c
C
b
B
t
A
P
t
A
P
c
c
c
b
d
c
b
d
c
b
d
c
b
d
c
b
d
c
b

What is P(C=true, A=true)?
A P(A)
false 0.6
true 0.4
A
B
C D
A B P(B|A)
false false 0.01
false true 0.99
true false 0.7
true true 0.3
B C P(C|B)
false false 0.4
false true 0.6
true false 0.9
true true 0.1
B D P(D|B)
false false 0.02
false true 0.98
true false 0.05
true true 0.95
18
.
0
45
.
0
*
4
.
0
)
42
.
0
03
.
0
(
4
.
0
)
1
*
6
.
0
*
7
.
0
1
*
1
.
0
*
3
.
0
(
4
.
0
))
|
(
)
|
(
)
|
(
)
|
(
)
|
(
)
|
(
(
4
.
0
)
|
(
)
|
(
)
|
(
)
(
)
|
(
)
|
(
)
|
(
)
(
)
,
,
,
(
)
,
(
,
,

















































f
B
d
D
P
f
B
t
C
P
t
A
f
B
P
t
B
d
D
P
t
B
t
C
P
t
A
t
B
P
b
B
d
D
P
b
B
t
C
P
t
A
b
B
P
t
A
P
b
B
d
D
P
b
B
t
C
P
t
A
b
B
P
t
A
P
d
D
t
C
b
B
t
A
P
t
C
t
A
P
d
d
d
b
d
b
d
b

33
The Bad News
• Exact inference is feasible in small to
medium-sized networks
• Exact inference in large networks takes a
very long time
• We resort to approximate inference
techniques which are much faster and give
pretty good results

34
One last unresolved issue…
We still haven’t said where we get the
Bayesian network from. There are two
options:
• Get an expert to design it
• Learn it from data, e.g., the same way as
in the lecture on Bayes Classifier in Ch. 8.

36
Sampling
Generate random samples and compute values of interest
from samples, not original network
• Input: Bayesian network with set of nodes X
• Sample = a tuple with assigned values
s=(X1=x1,X2=x2,… ,Xk=xk)
• Tuple may include all variables (except evidence E) or a
subset
• Sampling schemas dictate how to generate samples
(tuples)
• Ideally, samples are distributed according to P(X|E)

37
Sampling
• Idea: generate a set of samples T
• Estimate P(Xi|E) from samples
• Need to know:
– How to generate a new sample ?
– How many samples T do we need ?
– How to estimate P(Xi|E) ?

38
Sampling Algorithms
• Forward Sampling
• Likelyhood Weighting
• Gibbs Sampling (MCMC)
– Blocking
– Rao-Blackwellised
• Importance Sampling
• Sequential Monte-Carlo (Particle Filtering)
in Dynamic Bayesian Networks

39
Forward Sampling
• Forward Sampling
– Case with No evidence
– Case with Evidence
– N and Error Bounds

40
Forward Sampling No Evidence
(Henrion 1988)
Input: Bayesian network
X= {X1,…,XN}, N- #nodes, T - # samples
Output: T samples
Process nodes in topological order – first process
the ancestors of a node, then the node itself:
1. For t = 0 to T
2. For i = 0 to N
3. Xi  sample xi
t from P(xi | pai)

41
Sampling A Value
What does it mean to sample xi
t from P(Xi | pai) ?
• Assume D(Xi)={0,1}
• Assume P(Xi | pai) = (0.3, 0.7)
• Draw a random number r from [0,1]
If r falls in [0,0.3], set Xi = 0
If r falls in [0.3,1], set Xi=1
0 1
0.3 r

42
Sampling a Value
• When we sample xi
t from P(Xi | pai),
most of the time, will pick the most likely value of Xi
occasionally, will pick the unlikely value of Xi
• We want to find high-probability tuples
But!!!….
• Choosing unlikely value allows to “cross” the low
probability tuples to reach the high probability tuples !

43
Forward sampling (example)
1
X
2
X 3
X
4
X
)
( 1
x
P
)
|
( 1
2 x
x
P
)
,
|
( 3
2
4 x
x
x
P
)
|
( 1
3 x
x
P
)
|
(
from
sample
5.
otherwise
1,
from
start
and
sample
reject
0,
If
.
4
)
|
(
from
Sample
.
3
)
|
(
from
Sample
.
2
)
(
from
Sample
.
1
sample
generate
//
0
:
Evidence
3
,
2
4
4
3
1
3
3
1
2
2
1
1
3
x
x
x
P
x
x
x
x
P
x
x
x
P
x
x
P
x
k
X



44
Forward Sampling-Answering Queries
Task: given n samples {S1,S2,…,Sn}
estimate P(Xi = xi) :
T
x
X
samples
x
X
P i
i
i
i
)
(
#
)
(



Basically, count the proportion of samples where Xi = xi

45
Forward Sampling w/ Evidence
Input: Bayesian network
X= {X1,…,XN}, N- #nodes
E – evidence, T - # samples
Output: T samples consistent with E
1. For t=1 to T
2. For i=1 to N
3. Xi  sample xi
t from P(xi | pai)
4. If Xi in E and Xi  xi, reject sample:
5. i = 1 and go to step 2

46
Forward Sampling: Illustration
Let Y be a subset of evidence nodes s.t. Y=u

47
Gibbs Sampling
• Markov Chain Monte Carlo method
(Gelfand and Smith, 1990, Smith and Roberts, 1993, Tierney, 1994)
• Samples are dependent, form Markov Chain
• Samples directly from P(X|e)
• Guaranteed to converge when all P > 0
• Methods to improve convergence:
– Blocking
– Rao-Blackwellised

48
MCMC Sampling Fundamentals

 dx
X
x
g
g
E )
(
)
( 

Given a set of variables X = {X1, X2, … Xn} that
represent joint probability distribution (X) and some
function g(X), we can compute expected value of g(X) :

49
MCMC Sampling From (X)
Given independent, identically distributed samples
(iid) S1, S2, …ST from (X), it follows from Strong
Law of Large Numbers:
 

T
t
t
S
g
T
g 1
)
(
1
}
,...,
,
{ 2
1
t
n
t
t
t
x
x
x
S 
A sample St is an instantiation:

50
Gibbs Sampling (Pearl, 1988)
• A sample t[1,2,…],is an instantiation of all
variables in the network:
• Sampling process
– Fix values of observed variables e
– Instantiate node values in sample x0 at random
– Generate samples x1,x2,…xT from P(x|e)
– Compute posteriors from samples
}
,...,
,
{ 2
2
1
1
t
N
N
t
t
t
x
X
x
X
x
X
x 




51
Ordered Gibbs Sampler
Generate sample xt+1 from xt :
In short, for i=1 to N:
)
,

|
(
)
,
,...,
,
|
(
...
)
,
,...,
,
|
(
)
,
,...,
,
|
(
1
1
1
1
2
1
1
1
3
1
1
2
1
2
2
3
2
1
1
1
1
e
x
x
x
P
x
X
e
x
x
x
x
P
x
X
e
x
x
x
x
P
x
X
e
x
x
x
x
P
x
X
i
t
i
t
i
i
t
N
t
t
N
t
N
N
t
N
t
t
t
t
N
t
t
t
from
sampled

















Process
All
Variables
In Some
Order

52
Ordered Gibbs Sampling
Algorithm
Input: X, E
Output: T samples {xt }
• Fix evidence E
• Generate samples from P(X | E)
1. For t = 1 to T (compute samples)
2. For i = 1 to N (loop through variables)
3. Xi  sample xi
t from P(Xi | markovt Xi)
i
X

Answering Queries
• Query: P(xi |e)
• Method 1: count #of samples where Xi=xi:
Method 2: average probability (mixture estimator):
 



n
t i
t
i
i
i
i X
markov
x
X
P
T
x
X
P 1
)

|
(
1
)
(
T
x
X
samples
x
X
P i
i
i
i
)
(
#
)
(




54
Gibbs Sampling Example - BN
X = {X1,X2,…,X9}
E = {X9}
X1
X4
X8
X5
X2
X3
X9
X7
X6

55
X1 = x1
0
X6 = x6
0
X2 = x2
0
X7 = x7
0
X3 = x3
0
X8 = x8
0
X4 = x4
0
X5 = x5
0
X1
X4
X8
X5
X2
X3
X9
X7
X6

56
X1  P (X1 |X0
2,…,X0
8 ,X9}
E = {X9}
X1
X4
X8
X5
X2
X3
X9
X7
X6

57
X2  P(X2 |X1
1,…,X0
8 ,X9}
E = {X9}
X1
X4
X8
X5
X2
X3
X9
X7
X6

62
Gibbs Sampling: Illustration

63
Gibbs Sampling: Burn-In
• We want to sample from P(X | E)
• But…starting point is random
• Solution: throw away first K samples
• Known As “Burn-In”
• What is K ? Hard to tell. Use intuition.

64
Gibbs Sampling: Performance
+Advantage: guaranteed to converge to P(X|E)
-Disadvantage: convergence may be slow
Problems:
• Samples are dependent !
• Statistical variance is too big in high-dimensional
problems

65
Gibbs: Speeding Convergence
Objectives:
1. Reduce dependence between samples
(autocorrelation)
– Skip samples
– Randomize Variable Sampling Order
2. Reduce variance
– Blocking Gibbs Sampling
– Rao-Blackwellisation

66
Skipping Samples
• Pick only every k-th sample (Gayer, 1992)
Can reduce dependence between samples !
Increases variance ! Waists samples !

67
Randomized Variable Order
Random Scan Gibbs Sampler
Pick each next variable Xi for update at random
with probability pi , i pi = 1.
(In the simplest case, pi are distributed uniformly.)
In some instances, reduces variance (MacEachern,
Peruggia, 1999
“Subsampling the Gibbs Sampler: Variance Reduction”)

68
Blocking
• Sample several variables together, as a block
• Example: Given three variables X,Y,Z, with domains of
size 2, group Y and Z together to form a variable
W={Y,Z} with domain size 4. Then, given sample
(xt,yt,zt), compute next sample:
Xt+1  P(yt,zt)=P(wt)
(yt+1,zt+1)=Wt+1  P(xt+1)
+ Can improve convergence greatly when two variables are
strongly correlated!
- Domain of the block variable grows exponentially with
the #variables in a block!

69
Blocking Gibbs Sampling
Jensen, Kong, Kjaerulff, 1993
“Blocking Gibbs Sampling Very Large
Probabilistic Expert Systems”
• Select a set of subsets:
E1, E2, E3, …, Ek s.t. Ei  X
Ui Ei = X
Ai = X Ei
• Sample P(Ei | Ai)

70
Rao-Blackwellisation
• Do not sample all variables!
• Sample a subset!
• Example: Given three variables X,Y,Z,
sample only X and Y, sum out Z. Given
sample (xt,yt), compute next sample:
Xt+1  P(yt)
yt+1  P(xt+1)

71
Rao-Blackwell Theorem
Bottom line: reducing number of variables in a sample reduce variance!

73
Geman&Geman1984
• Geman, S. & Geman D., 1984. Stocahstic relaxation,
Gibbs distributions, and the Bayesian restoration of
images. IEEE Trans.Pat.Anal.Mach.Intel. 6, 721-41.
– Introduce Gibbs sampling;
– Place the idea of Gibbs sampling in a general setting in
which the collection of variables is structured in a
graphical model and each variable has a neighborhood
corresponding to a local region of the graphical structure.
Geman and Geman use the Gibbs distribution to define the
joint distribution on this structured set of variables.

74
Tanner&Wong 1987
• Tanner and Wong (1987)
– Data-augmentation
– Convergence Results

75
Pearl1988
• Pearl,1988. Probabilistic Reasoning in
Intelligent Systems, Morgan-Kaufmann.
– In the case of Bayesian networks, the
neighborhoods correspond to the Markov
blanket of a variable and the joint distribution is
defined by the factorization of the network.

76
Gelfand&Smith,1990
• Gelfand, A.E. and Smith, A.F.M., 1990.
Sampling-based approaches to calculating
marginal densities. J. Am.Statist. Assoc. 85,
398-409.
– Show variance reduction in using mixture
estimator for posterior marginals.

77
Neal, 1992
• R. M. Neal, 1992. Connectionist
learning of belief networks, Artifical
Intelligence, v. 56, pp. 71-118.
– Stochastic simulation in noisy-or networks.

Bayesian Networks Model in Step By Steps

More Related Content

Similar to Bayesian Networks Model in Step By Steps (20)

Recently uploaded (20)

Bayesian Networks Model in Step By Steps