Bayesian intro

An Introduction to the
Bayesian Approach

J Guzmán, PhD
15 August 2011

Bayesian: one who asks you
what you think before a study in
order to tell you what you think
afterwards

Adapted from:
S Senn (1997). Statistical Issues in
Drug Development. Wiley

Rev. Thomas Bayes

English Theologian and
Mathematician

ca. 1700 – 1761

Bayesian Methods
•  1763 – Bayes’ article on inverse probability
•  Laplace extended Bayesian ideas in different
scientific areas in Théorie Analytique des
Probabilités [1812]
•  Both Laplace & Gauss used the inverse method
•  1st three quarters of 20th Century dominated by
frequentist methods
•  Last quarter of 20th Century – resurgence of
Bayesian methods [computational advances]
•  21st Century – Bayesian Century [Lindley]

Pierre-Simon Laplace

French Mathematician

1749 – 1827

Karl Friedrich Gauss

“Prince of Mathematics”

1777 – 1855

Used inverse probability

Bayesian Methods
•  Key components: prior, likelihood function,
posterior, and predictive distribution
•  Suppose a study is carried out to compare new
and standard teaching methods
•  Ho: Methods are equally effective
•  HA: New method increases grades by 20%
•  A Bayesian presents the probability that new &
standard methods are equally effective, given
the results of the experiment at hand:
P(Ho | data)

Bayesian Methods
•  Data – observed data from experiment

•  Find the probability that the new method is at least 20%
more effective than the standard, given the results of the
experiment [Posterior Probability]

•  Another conclusion could be the probability distribution
for the outcome of interest for the next student

•  Predictive Probabilities – refer to future observations on
individuals or on set of individuals

Bayes’ Theorem
•  Basic tool of Bayesian analysis
•  Provide the means by which we learn from
data
•  Given prior state of knowledge, it tells how
to update belief based upon observations:
∝

P(H | data) = P(H) · P(data | H) / P(data)
€

α P(H) · P(data | H)
∝

α means “is proportonal to”
•  Bayes’ theorem can be re-expressed in
€

odds terms: let data ≡ y

Bayes’ Theorem
•  Can also consider posterior probability of
any measure θ:
P(θ | data) α P(θ) · P( data | θ)
•  Bayes’ theorem states that the posterior
probability of any measure θ, is
proportional to the information on θ
external to the experiment times the
likelihood function evaluated at θ:
Prior · likelihood → posterior

Prior
•  Prior information about θ assessed as a
probability distribution on θ
•  Distribution on θ depends on the assessor: it is
subjective
•  A subjective probability can be calculated any
time a person has an opinion
•  Diffuse prior - when a person’ s opinion on θ
includes a broad range of possibilities & all
values are thought to be roughly equally
probable

Prior
•  Conjugate prior – if the posterior distribution
has same shape as the prior distribution,
regardless of the observed sample values
•  Examples:
1.  Beta prior & binomial likelihood yield a beta posterior
2.  Normal prior & normal likelihood yield a normal
posterior
3.  Gamma prior & Poisson likelihood yield a gamma
posterior

Community of Priors
•  Expressing a range of reasonable opinions
•  Reference – represents minimal prior
information
•  Expertise – formalizes opinion of well-
informed experts
•  Skeptical – downgrades superiority of new
method
•  Enthusiastic – counterbalance of skeptical

Likelihood Function
P(data | θ)
•  Represents the weighting of evidence from the
experiment about θ
•  It states what the experiment says about the
measure of interest [Savage, 1962]
•  It is the probability of getting certain result,
conditioning on the model
•  As the amount of data increases, Prior is
dominated by the Likelihood :
–  Two investigators with different prior opinions
could reach a consensus after the results of an
experiment

Likelihood Principle
•  States that the likelihood function contains
all relevant information from the data
•  Two samples have equivalent information
if their likelihoods are proportional
•  Adherence to the Likelihood Principle
means that inference are conditional on
the observed data
•  Bayesian analysts base all inferences
about θ solely on its posterior distribution

Likelihood Principle
•  Two experiments: one yields data y1 and
the other yields data y2
•  If the likelihoods: P(y1 | θ) & P(y2 | θ) are
identical up to multiplication by arbitrary
functions of y1 & y2 then they contain
identical information about θ and lead to
identical posterior distributions
•  Therefore, to equivalent inferences

Example
•  EXP 1: In a study of a •  EXP 2: Students are
fixed sample of 20 entered into a study
students, 12 of them until 12 of them
respond positively to respond positively to
the method [Binomial the method [Negative-
distribution] binomial distribution]

•  Likelihood is •  Likelihood at n = 20 is
proportional to proportional to
θ12 (1 – θ)8 θ12 (1 – θ)8

Exchangeability
•  Key idea in statistical inference in general
•  Two observations are exchangeable if they provide
equivalent statistical information
•  Two students randomly selected from a particular
population of students can be considered
exchangeable
•  If the students in a study are exchangeable with the
students in the population for which the method is
intended, then the study can be used to make
inferences about the entire population
•  Exchangeability in terms of experiments: Two studies
are exchangeable if they provide equivalent statistical
information about some super-population of
experiments

Laplace on Probability
It is remarkable that a science, which
commenced with the consideration of
games of chance, should be elevated to
the rank of the most important subjects
of human knowledge.
A Philosophical Essay on Probabilities.
John Wiley & Sons, 1902, page 195.
Original French edition 1814.

References
•  Computation:
OpenBUGS http://mathstat.helsinki.fi/openbugs/
R packages: BRugs, bayesm, R2WinBUGS from CRAN: http://
cran.r-project.org/
•  Gelman, A, Carlin, JB, Stern, HS, & Rubin, DB (2004). Bayesian
Data Analysis. Second Ed.. Chapman and Hall
•  Gilks, WR, Richardson, S, & Spiegelhalter, DJ (1996). Markov Chain
Monte Carlo in Practice. Chapman & Hall

•  More Advanced:
Bernardo, J & Smith, AFM (1994). Bayesian Theory. Wiley
O'Hagan, A & Forster, JJ (2004). Bayesian Inference, 2nd Edition.
Vol. 2B of "Kendall's Advanced Theory of Statistics". Arnold

Bayesian intro

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Bayesian intro (20)

Bayesian intro