2.03 bayesian estimation

1. Introduction to Mathematics for AI Bayesian Estimation Andres Mendez-Vazquez May 22, 2020 1 / 117

2. Outline 1 Introduction Likelihood Principle Example, Testing Fairness Independence Influence Sufficiency Fisher-Neyman Characterization Example Sufficiency Principle Conditionality Perspective Example Sins of Being non-Bayesian 2 Bayesian Inference Introduction Connection with Sufficient Statistics Generalized Maximum Likelihood Estimator The Maximum A Posteriori (MAP) Maximum Likelihood Vs Maximum A Posteriori Properties of the MAP 3 Loss, Posterior Risk, Bayes Action Bayes Principle in the Frequentist Decision Theoretic Setup Examples of Loss Functions Bayesian Expected Loss Principle Example The Empirical Risk The Fubini’s Theorem 2 / 117

4. The Basis of Bayesian Inference A Basic setup Let f(x|θ) be a conditional distribution for X given the unknown parameter θ. For the observed data, X = x, the function (θ) = f(x|θ) It is called the likelihood function!!! The name likelihood implies that, given x, the value of θ It is more likely to be the true parameter than θ , if f (x|θ) > f x|θ 4 / 117

7. Basically We are talking about optimization functions Where optimals are being loked upon... Deﬁnition An optimal solution to an optimization poroblems is the feasible solution with the largest objective function value (for a maximization problem). 5 / 117

8. Basically We are talking about optimization functions Where optimals are being loked upon... Deﬁnition An optimal solution to an optimization poroblems is the feasible solution with the largest objective function value (for a maximization problem). 5 / 117

9. Likelihood Principle Remarks In the inference about θ, after x is observed, all relevant experimental information is contained in the likelihood function for the observed x. There is an interesting example quoted by Lindley and Phillips in 1976 [1] Originally by Leonard Savage Leonard Savage Leonard Jimmie Savage (born Leonard Ogashevitz; 20 November 1917 – 1 November 1971) was an American mathematician and statistician. Economist Milton Friedman said Savage was "one of the few people I have met whom I would unhesitatingly call a genius. 6 / 117

12. History Something Notable The likelihood principle was ﬁrst identiﬁed by that name in print in 1962 (Barnard et al., Birnbaum, and Savage et al.), However Fisher It was already using a version of it in 1920’s. However, versions of it can be tracked to To the mid-1700s It seems to have become a commonplace among natural philosophers that problems of observational error were susceptible to mathematical description. 7 / 117

16. Testing Fairness Basic Setup Suppose we are interested in testing θ, the unknown probability of heads for possibly biased coin. Suppose the following Hypothesis H0 : θ = 1/2 v.s. H1 : θ > 1/2 Then An experiment is conducted and 9 heads and 3 tails are observed. Not enough information to fully specify f (x|θ) 9 / 117

19. Scenario 1 Based on rashomonian analysis The classic Akira Kurosawa ﬁlm Rashomon has become a shorthand for the lie of objective truth—what you see, basically, depends on where you stand. Number of ﬂips, n = 12 is predetermined Then number of heads X is binomial B(n, θ), with probability mass function: Pθ (X = x) = f (x|θ) = n x θx (1 − θ)n−x 10 / 117

20. Scenario 1 Based on rashomonian analysis The classic Akira Kurosawa ﬁlm Rashomon has become a shorthand for the lie of objective truth—what you see, basically, depends on where you stand. Number of ﬂips, n = 12 is predetermined Then number of heads X is binomial B(n, θ), with probability mass function: Pθ (X = x) = f (x|θ) = n x θx (1 − θ)n−x 10 / 117

21. Therefore We have Pθ (X = x) = 12 9 θ9 (1 − θ)3 Thus We can use the p − value for testing the hypothesis. 11 / 117

22. Therefore We have Pθ (X = x) = 12 9 θ9 (1 − θ)3 Thus We can use the p − value for testing the hypothesis. 11 / 117

23. Then if we use the following p − value analysis Deﬁnition [2, 3] The p-value is deﬁned as the probability, under the null hypothesis H0 about the unknown distribution F of the random variable X. Very Unlike Observation Very Unlike Observation P-Value More Likely Observation Set of Possible Results ProbabilityDensity Observed Data Point 12 / 117

24. Therefore For a frequentist, the p − value of the test is P (X ≥ 9|H0) = 12 x=9 12 x 1 2 x 1 2 12−x = 0.073 Given an α = 0.05 Then, H0 is not rejected... 13 / 117

25. Therefore For a frequentist, the p − value of the test is P (X ≥ 9|H0) = 12 x=9 12 x 1 2 x 1 2 12−x = 0.073 Given an α = 0.05 Then, H0 is not rejected... 13 / 117

26. Scenario 2 Number of tails (successes) 3 is predetermined i.e, the ﬂipping is continued until 3 tails are observed. Then you have a Negative Binomial with r the number of failures f (x|θ) = k + r − 1 k − 1 (1 − θ)k θr Thus, we have f (x|θ) = 3 + 9 − 1 3 − 1 (1 − θ)3 θ9 = 55 (1 − θ)3 θ9 14 / 117

29. In a similar way We have P (X ≥ 9|H0) = ∞ x=9 3 + x − 1 3 − 1 1 2 x 1 2 3 = 0.0327 Thus, the hypothesis H0 is rejected But this change in decision is not caused by observations. However, all relevant information is in the likelihood!!! (θ) ∝ θ9 (1 − θ)3 15 / 117

32. Remark Edwards, Lindman, and Savage remarked The likelihood principle emphasized in Bayesian statistics implies, among other things, that the rules governing when data collection stops are irrelevant to data interpretation. Therefore It is entirely appropriate to collect data until a point has been proven or disproven, or until the data collector runs out of time, money, or patience. 16 / 117

33. Remark Edwards, Lindman, and Savage remarked The likelihood principle emphasized in Bayesian statistics implies, among other things, that the rules governing when data collection stops are irrelevant to data interpretation. Therefore It is entirely appropriate to collect data until a point has been proven or disproven, or until the data collector runs out of time, money, or patience. 16 / 117

34. Thus Likelihood Principle [4] The Likelihood principle (LP) asserts that for inference on an unknown quantity θ, all of the evidence from any observation X = x with distribution X ∼ f (x|θ) lies in the likelihood function L (θ|x) ∝ f (x|θ) , θ ∈ Θ 17 / 117

35. Thus Something Notable The interpretation of LP hinges on the rather subtle point of allowing any observable X to draw conclusions about θ. Therefore If there two ways to gather infromation about theta, wither X ∼ f (x|θ) or with Y ∼ g (x|θ) with X = x and Y = y then L (θ|x) = η × L (θ|y) , ∀θ ∈ Θ 18 / 117

36. Thus Something Notable The interpretation of LP hinges on the rather subtle point of allowing any observable X to draw conclusions about θ. Therefore If there two ways to gather infromation about theta, wither X ∼ f (x|θ) or with Y ∼ g (x|θ) with X = x and Y = y then L (θ|x) = η × L (θ|y) , ∀θ ∈ Θ 18 / 117

38. In the case of Learning Yes, we use the principle, but we add the idea of independence A trick to assume a set of samples x1, x2, ..., xN such that xi ∼ f (X|θ) Then, as we have seen it L (θ) = f (x1, x2, ..., xN |θ) = N i=1 f (xi|θ) 20 / 117

39. In the case of Learning Yes, we use the principle, but we add the idea of independence A trick to assume a set of samples x1, x2, ..., xN such that xi ∼ f (X|θ) Then, as we have seen it L (θ) = f (x1, x2, ..., xN |θ) = N i=1 f (xi|θ) 20 / 117

40. Example, p (x|ωj) ∼ N µj, Σj L (θj) = log n j=1 p (xj|θj) 21 / 117

42. The Basics Sufficiency Principle An statistic is sufficient with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter"[5] However, as always We want a definition to build upon it... as always 23 / 117

43. The Basics Sufficiency Principle An statistic is sufficient with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter"[5] However, as always We want a definition to build upon it... as always 23 / 117

44. A Basic Definition Definition A statistic t = T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ [6]. Something Notable This agreement is non-philosophical, it is rather a consequence of mathematics (measure theoretic considerations). 24 / 117

45. A Basic Definition Definition A statistic t = T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ [6]. Something Notable This agreement is non-philosophical, it is rather a consequence of mathematics (measure theoretic considerations). 24 / 117

47. Fisher’s Factorization Theorem Theorem Let f (x|θ) be the density or mass function for the random vector x, parametrized by the vector theta. The statistic t = T (x) is suﬃcient for θ if and only if there exist functions a (x) (not depending on θ) and b (t|θ) such that () f (x|θ) = a (x) b (t, θ) for all possible values of x. 26 / 117

53. Now, for the case ⇐ Suppose the probability mass function for x can be written f (x|θ) = a (x) b (x|θ) where t = T (x) The probability mass function for t is obtained by summing fθ (x, t) over all x such that T (x) = t f (t|θ) = T(x)=t f (x, t|θ) = T(x)=t f (x|θ) ← independence over t = T(x)=t a (x) bθ (x) 28 / 117

57. Therefore, we have that The conditional mass function of x given t f (x|θ, t) = f (x, t|θ) f (t|θ) = f (x|θ) f (t|θ) = a (x) bθ (x) T(x)=t a (x) bθ (x) = a (x) T(x)=t a (x) This last expression does not depend on θ t is a suﬃcient statistic for θ. 29 / 117

58. Therefore, we have that The conditional mass function of x given t f (x|θ, t) = f (x, t|θ) f (t|θ) = f (x|θ) f (t|θ) = a (x) bθ (x) T(x)=t a (x) bθ (x) = a (x) T(x)=t a (x) This last expression does not depend on θ t is a suﬃcient statistic for θ. 29 / 117

60. Using the Bernoulli Distribution xn ∼Bernoulli(θ) are i.d.d. ∀n = 1, ..., N f (x1, .., xN |θ) = N n=1 θxn (1 − θ)1−xn = θk (1 − θ)N−k k = N n=1 xn Now, if we have the following choices a (x) = 1 and bθ (k) = θk (1 − θ)N−k 31 / 117

61. Using the Bernoulli Distribution xn ∼Bernoulli(θ) are i.d.d. ∀n = 1, ..., N f (x1, .., xN |θ) = N n=1 θxn (1 − θ)1−xn = θk (1 − θ)N−k k = N n=1 xn Now, if we have the following choices a (x) = 1 and bθ (k) = θk (1 − θ)N−k 31 / 117

62. Therefore Then choosing T (x1, .., xN ) = N n=1 xn = k By the Fisher-Neyman Factorization Theorem k is suﬃcient for θ 32 / 117

64. Something Quite Interesting The Fisher-Neyman factorization lemma states The likelihood can be represented as (θ) = f (x|θ) = a (x) bθ (T (x)) 34 / 117

65. If the likelihood principle is adopted All inference about θ should depend on sufficient statistics Since (θ) ∝ bθ (T (x)) Sufficiency Principle Let the two different observations x and y have the same values T (x) = T (y), of a statistics sufficient for family f (·|θ). Then the inferences about θ based on x and y should be the same. 35 / 117

66. If the likelihood principle is adopted All inference about θ should depend on sufficient statistics Since (θ) ∝ bθ (T (x)) Sufficiency Principle Let the two different observations x and y have the same values T (x) = T (y), of a statistics sufficient for family f (·|θ). Then the inferences about θ based on x and y should be the same. 35 / 117

68. Conditionality Perspective We have that Conditional perspective concerns reporting data speciﬁc measures of accuracy. In contrast to the frequentist approach Performance of statistical procedures are judged looking at the observed data. 37 / 117

69. Conditionality Perspective We have that Conditional perspective concerns reporting data speciﬁc measures of accuracy. In contrast to the frequentist approach Performance of statistical procedures are judged looking at the observed data. 37 / 117

71. Example Consider estimating θ in the model P (X = θ − 1|θ) = P (X = θ + 1|θ) with θ ∈ R on basis of two observations, X1 and X2 . The procedure suggested is δ (X) = X1+X2 2 if X1 = X2 X1 − 1 if X1 = X2 39 / 117

72. Example Consider estimating θ in the model P (X = θ − 1|θ) = P (X = θ + 1|θ) with θ ∈ R on basis of two observations, X1 and X2 . The procedure suggested is δ (X) = X1+X2 2 if X1 = X2 X1 − 1 if X1 = X2 39 / 117

73. Therefore To a frequentist, this procedure has confidence To a frequentist, this procedure has confidence of 75% for all θ, i.e., P (δ (X) = θ) = 0.75. The conditionalist would report the confidence 100% if observed data in hand are different 50% if the observations coincide 40 / 117

74. Therefore To a frequentist, this procedure has confidence To a frequentist, this procedure has confidence of 75% for all θ, i.e., P (δ (X) = θ) = 0.75. The conditionalist would report the confidence 100% if observed data in hand are different 50% if the observations coincide 40 / 117

75. Then Conditionality Principle If an experiment concerning the inference about θ is chosen from a collection of possible experiments, independently of θ, then any experiment not chosen is irrelevant to the inference. 41 / 117

77. Not a good idea to integrate with respect to sample space What? A perfectly valid hypothesis can be rejected because the test failed to account for unlikely data that had not been observed... 43 / 117

78. The Lindley Paradox Suppose y|θ ∼ N θ, 1 n We wish to test H0 : θ = 0 vs the two sided alternative. Suppose a Bayesian puts the prior P (θ = 0) = P (θ = 0) = 1 2 The 1 2 is uniformly spread over the interval [−M/2, M/2]. Suppose n = 40, 000 and y = 0.01 are observed So, √ ny = 2 44 / 117

81. Therefore Classical statistician She/he rejects H0 at level α = 0.05 Posterior odds in favor of H0 are 11 if M = 1 We will look at this... no worries, but Bayesian Statistician will choose H0 45 / 117

82. Therefore Classical statistician She/he rejects H0 at level α = 0.05 Posterior odds in favor of H0 are 11 if M = 1 We will look at this... no worries, but Bayesian Statistician will choose H0 45 / 117

84. Using our likelihood We have our function (θ) = f (x|θ) Here The parameter θ is supported by the parameter space Θ and considered a random variable. The random variable θ has a distribution π (θ) that is called the prior. 47 / 117

85. Using our likelihood We have our function (θ) = f (x|θ) Here The parameter θ is supported by the parameter space Θ and considered a random variable. The random variable θ has a distribution π (θ) that is called the prior. 47 / 117

86. Not only that We have the following We can play a hierarchy game θ ∼ π (θ|τ) where τ is called a hyperparameter This give us an idea about the marginals m (x) = Θ f (x, θ) = Θ f (x|θ) π (θ) dθ 48 / 117

87. Not only that We have the following We can play a hierarchy game θ ∼ π (θ|τ) where τ is called a hyperparameter This give us an idea about the marginals m (x) = Θ f (x, θ) = Θ f (x|θ) π (θ) dθ 48 / 117

88. What about the posterior? We have the following f (θ|x) = f (x, θ) m (x) = f (x|θ) π (θ) m (x) = f (x|θ) π (θ) Θ f (x|θ) π (θ) dθ 49 / 117

92. An interesting case Suppose that the observations are coming from N (θ, σ2 1) Assume prior on θ is N (σ2, σ2) Then, under this setup the normal/normal model, the posterior is f (θ|X1, ..., Xn) = f θ|X 51 / 117

93. An interesting case Suppose that the observations are coming from N (θ, σ2 1) Assume prior on θ is N (σ2, σ2) Then, under this setup the normal/normal model, the posterior is f (θ|X1, ..., Xn) = f θ|X 51 / 117

94. The connection Lemma Suppose the suﬃcient statistics T = T (X1, ..., Xn) exist. Then f (θ|X1, ..., Xn) = f (θ|T) . 52 / 117

95. Proof Factorization theorem for suﬃcient statistics is f (x|θ) = bθ (t) a (x) Where t = T (x) and a (x) do not depend on θ. 53 / 117

96. Proof Factorization theorem for suﬃcient statistics is f (x|θ) = bθ (t) a (x) Where t = T (x) and a (x) do not depend on θ. 53 / 117

97. Furhtermore Thus π (θ|x) = f (x|θ) π (θ) Θ f (x|θ) π (θ) dθ = bθ (t) a (x) π (θ) Θ bθ (t) a (x) π (θ) dθ = bθ (t) π (θ) Θ bθ (t) π (θ) dθ 54 / 117

100. The Multiply and divide by φ (t) = bθ (t) π (θ) φ (t) Θ bθ (t) π (θ) φ (t) dθ = bθ (t) π (θ) φ (t) Θ bθ (t) π (θ) φ (t) dθ = π (θ) f (t|θ) Θ π (θ) f (t|θ) dθ = π (θ|t) 55 / 117

103. Here, we have The following equations f (t|θ) = x:T(x)=t f (x|θ) dx = x:T(x)=t bθ (t) a (x) dx Then x:T(x)=t bθ (t) a (x) dx = bθ (t) x:T(x)=t a (x) dx = bθ (t) φ (t) 56 / 117

104. Here, we have The following equations f (t|θ) = x:T(x)=t f (x|θ) dx = x:T(x)=t bθ (t) a (x) dx Then x:T(x)=t bθ (t) a (x) dx = bθ (t) x:T(x)=t a (x) dx = bθ (t) φ (t) 56 / 117

105. Then We have the following definition Definition The statistics T = T(X) is sufficient (in the Bayesian sense) if for any prior the resulting posterior satisfies π (θ|X) = π (θ|T) This is equivalent to the classic definition on sufficient statistics Theorem T is sufficient in the Bayesian sense if and only if it is sufficient in the usual sense. 57 / 117

106. Then We have the following definition Definition The statistics T = T(X) is sufficient (in the Bayesian sense) if for any prior the resulting posterior satisfies π (θ|X) = π (θ|T) This is equivalent to the classic definition on sufficient statistics Theorem T is sufficient in the Bayesian sense if and only if it is sufficient in the usual sense. 57 / 117

108. Something quite important Something Notable The posterior is the ultimate experimental summary for a Bayesian. Not only that The location measures (especially the mean) of the posterior are of importance. There is an important idea The posterior mode and median are also Bayes estimators under diﬀerent loss functions!!! 59 / 117

111. Furthermore Generalized Maximum Likelihood Estimator AKA MAP (Maximum Aposteriori) The generalized MLE is the largest mode of the π (θ|x). 60 / 117

113. What can we do? We can specify a distribution Then, learn the parameters Remember the Bayesian Rule p (Θ|X) = p (X|Θ) p (Θ) p (X) (1) We seek that value for Θ, called ΘMAP It allows to maximize the posterior p (Θ|X) 62 / 117

116. Therefore We can use this idea of maximizing the posterior To obtain the distribution through the Maximum a Posteriori 63 / 117

117. Development of the solution We look to maximize ΘMAP ΘMAP = argmax Θ p (Θ|X) = argmax Θ p (X|Θ) p (Θ) P (X) ≈ argmax Θ p (X|Θ) p (Θ) = argmax Θ xi∈X p (xi|Θ) p (Θ) P (X) can be removed because it has no functional relation with Θ. 64 / 117

122. We can make this easier Use logarithms ΘMAP = argmax Θ   xi∈X log p (xi|Θ) + log p (Θ)   (2) 65 / 117

123. What Does the MAP Estimate Get? Something Notable The MAP estimate allows us to inject into the estimation calculation our prior beliefs regarding the parameters values in Θ. For example Let’s conduct N independent trials of the following Bernoulli experiment with q parameter: We will ask each individual we run into in the hallway whether they will vote PRI or PAN in the next presidential election. With probability q to vote PRI Where the values of xi is either PRI or PAN. 66 / 117

128. First the Maximum Likelihood Estimate Samples X = xi = PAN PRI i = 1, ..., N (3) The log likelihood function log p (X|q) = N i=1 log p (xi|q) = i log p (xi = PRI|q) + ... i log p (xi = PAN|1 − q) =nPRI log (q) + (N − nPRI) log (1 − q) Where nPRI are the numbers of individuals who are planning to vote PRI this fall 68 / 117

133. We use our classic tricks By setting L = log p (X|q) (4) We have that ∂L ∂q = 0 (5) Thus nPRI q − (N − nPRI) (1 − q) = 0 (6) 69 / 117

136. Final Solution of ML We get qPRI = nPRI N (7) Thus If we say that N = 20 and if 12 are going to vote PRI, we get qPRI = 0.6. 70 / 117

137. Final Solution of ML We get qPRI = nPRI N (7) Thus If we say that N = 20 and if 12 are going to vote PRI, we get qPRI = 0.6. 70 / 117

138. Building the MAP estimate Obviously we need a prior belief distribution We have the following constraints: The prior for q must be zero outside the [0, 1] interval. Within the [0, 1] interval, we are free to specify our beliefs in any way we wish. In most cases, we would want to choose a distribution for the prior beliefs that peaks somewhere in the [0, 1] interval. We assume the following The state of Colima has traditionally voted PRI in presidential elections. However, on account of the prevailing economic conditions, the voters are more likely to vote PAN in the election in question. 71 / 117

144. What prior distribution can we use? We could use a Beta distribution being parametrized by two values α and β p (q) = 1 B (α, β) qα−1 (1 − q)β−1 . (8) Where We have B (α, β) = Γ(α)Γ(β) Γ(α+β) is the beta function where Γ is the generalization of the notion of factorial in the case of the real numbers. Properties When both the α, β > 0 then the beta distribution has its mode (Maximum value) at α − 1 α + β − 2 . (9) 72 / 117

147. We then do the following We do the following We can choose α = β so the beta prior peaks at 0.5. As a further expression of our belief We make the following choice α = β = 5. Why? Look at the variance of the beta distribution αβ (α + β)2 (α + β + 1) . (10) 73 / 117

150. Thus, we have the following nice properties We have a variance with α = β = 5 V ar (q) ≈ 0.025 Thus, the standard deviation sd ≈ 0.16 which is a nice dispersion at the peak point!!! 74 / 117

151. Thus, we have the following nice properties We have a variance with α = β = 5 V ar (q) ≈ 0.025 Thus, the standard deviation sd ≈ 0.16 which is a nice dispersion at the peak point!!! 74 / 117

152. Now, our MAP estimate for pMAP ... We have then pMAP = argmax Θ   xi∈X log p (xi|q) + log p (q)   (11) Plugging back the ML pMAP = argmax Θ [nPRI log q + (N − nPRI) log (1 − q) + log p (q)] (12) Where log p (q) = log 1 B (α, β) qα−1 (1 − q)β−1 (13) 75 / 117

155. The log of p (q) We have that log p (q) = (α − 1) log q + (β − 1) log (1 − q) − log B (α, β) (14) Now taking the derivative with respect to q, we get nPRI q − (N − nPRI) (1 − q) − β − 1 1 − q + α − 1 q = 0 (15) Thus qMAP = nPRI + α − 1 N + α + β − 2 (16) 76 / 117

158. Now With N = 20 with nPRI = 12 and α = β = 5 qMAP = 0.571 77 / 117

159. Another Example Let X1, ..., Xn given θ are Poisson P (θ) with probability f (xi|θ) = θxi xi! e−θ Assume θ ∼ Γ (α, β) given by π (θ) ∝ θα−1e−βθ The MAP is equal to π (θ|X1, X2, ..., Xn) = π θ| Xi ∝ θ Xi+α−1 e−(n+β)θ Basically Γ ( Xi + α − 1, n + β) The mean is E [θ|X] = Xi + α n + β 78 / 117

162. Now, given the mean of the Γ We can rewrite the mean as E [θ|X] = n n + β × Xi n + β β + n × α β Given that the means are Mean of MLE Xi n Mean of the prior α β 79 / 117

163. Now, given the mean of the Γ We can rewrite the mean as E [θ|X] = n n + β × Xi n + β β + n × α β Given that the means are Mean of MLE Xi n Mean of the prior α β 79 / 117

164. Remarks Something Notable The standard MLE maximizes π (θ|x), while the generalized MLE maximizes π(θ) (θ). Quite funny we call that Maximum Aposteriori (MAP) estimator!!! The MAP estimator is since it is often simpler to calculate given that arg max θ π (θ|x) = arg max θ f (x|θ) π (θ) Given that the normalization factor is a constant 80 / 117

165. Remarks Something Notable The standard MLE maximizes π (θ|x), while the generalized MLE maximizes π(θ) (θ). Quite funny we call that Maximum Aposteriori (MAP) estimator!!! The MAP estimator is since it is often simpler to calculate given that arg max θ π (θ|x) = arg max θ f (x|θ) π (θ) Given that the normalization factor is a constant 80 / 117

167. Properties First MAP estimation “pulls” the estimate toward the prior. Second The more focused our prior belief, the larger the pull toward the prior. Example If α = β =equal to large value It will make the MAP estimate to move closer to the prior. 82 / 117

170. Properties Third In the expression we derived for qMAP , the parameters α and β play a “smoothing” role vis-a-vis the measurement nPRI. Fourth Since we referred to q as the parameter to be estimated, we can refer to α and β as the hyper-parameters in the estimation calculations. 83 / 117

171. Properties Third In the expression we derived for qMAP , the parameters α and β play a “smoothing” role vis-a-vis the measurement nPRI. Fourth Since we referred to q as the parameter to be estimated, we can refer to α and β as the hyper-parameters in the estimation calculations. 83 / 117

172. Beyond simple derivation In the previous technique We took an logarithm of the likelihood × the prior to obtain a function that can be derived in order to obtain each of the parameters to be estimated. What if we cannot derive? For example when we have something like |θi|. We can try the following EM + MAP to be able to estimate the sought parameters. 84 / 117

176. Imagine an action space and a ∈ A For example In estimation problems, A is the set of real numbers and a is a number, say a = 2 is adopted as an estimator of θ ∈ Θ. Another One In testing problems, the action space is A = {accept, reject} 86 / 117

177. Imagine an action space and a ∈ A For example In estimation problems, A is the set of real numbers and a is a number, say a = 2 is adopted as an estimator of θ ∈ Θ. Another One In testing problems, the action space is A = {accept, reject} 86 / 117

178. Everytime you make a decision you have a Loss Actually Statisticians are pessimistic creatures that replaced nicely coined term utility to a more somber term loss!!! How do we denote such losses? A classic one L (θ, a) representing the payoﬀ by a decision maker (statistician) if he takes any action a ∈ A in certina state of nature θ 87 / 117

179. Everytime you make a decision you have a Loss Actually Statisticians are pessimistic creatures that replaced nicely coined term utility to a more somber term loss!!! How do we denote such losses? A classic one L (θ, a) representing the payoﬀ by a decision maker (statistician) if he takes any action a ∈ A in certina state of nature θ 87 / 117

181. Examples Squared Error Loss L (θ, a) = (θ − a)2 Absolute Loss L (θ, a) = |θ − a| 0-1 Loss example L (θ, a) = I [|θ − a| > m] 89 / 117

184. Clearly the easiest mathematically SEL Additionally, it is linked with EX|θ [θ − δ (X)]2 = V ar (δ (X)) + [bias (δ (X))]2 Where bias (δ (X)) = EX|θ [δ (X)] − θ 90 / 117

189. Then Using the following equivalence ∂ ∂x g(x) f(x) φ (x, t) dt = g(x) f(x) ∂ ∂x φ (x, t) dt + φ (x, g (x)) ∂g (x) ∂x − ... φ (x, f (x)) ∂f (x) ∂x Then ∂ϕ (a) ∂a = − ∞ a π (θ|X) dθ + 0 − 0 + a ∞ π (θ|X) dθ + 0 − 0 92 / 117

190. Then Using the following equivalence ∂ ∂x g(x) f(x) φ (x, t) dt = g(x) f(x) ∂ ∂x φ (x, t) dt + φ (x, g (x)) ∂g (x) ∂x − ... φ (x, f (x)) ∂f (x) ∂x Then ∂ϕ (a) ∂a = − ∞ a π (θ|X) dθ + 0 − 0 + a ∞ π (θ|X) dθ + 0 − 0 92 / 117

193. Finally The Median Minimize ϕ (a) 94 / 117

195. Bayesian Expected Loss Deﬁnition Bayesian expected loss is the expectation of the loss function with respect to posterior measure, ρ (a, π) = Eθ|X [L (a, θ)] = Θ L (θ, a) π (θ|x) dθ Here, we have an important principle Referring to the less possible loss!!! 96 / 117

196. Bayesian Expected Loss Deﬁnition Bayesian expected loss is the expectation of the loss function with respect to posterior measure, ρ (a, π) = Eθ|X [L (a, θ)] = Θ L (θ, a) π (θ|x) dθ Here, we have an important principle Referring to the less possible loss!!! 96 / 117

197. The Expected Loss Principle Deﬁnition In comparing two actions a1 = δ1(X) and a2 = δ2(X), after data X had been observed, preferred action is the one for which the posterior expected loss is smaller. Therefore An action a∗ that minimizes the posterior expected loss is called Bayes action. 97 / 117

198. The Expected Loss Principle Deﬁnition In comparing two actions a1 = δ1(X) and a2 = δ2(X), after data X had been observed, preferred action is the one for which the posterior expected loss is smaller. Therefore An action a∗ that minimizes the posterior expected loss is called Bayes action. 97 / 117

200. Example If the loss is squared error The Bayes action a∗ is found by minimizing ϕ (a) = Eθ|X (θ − a)2 = a2 − 2Eθ|X [θ] a + Eθ|Xθ2 Then, we want ϕ (a) = 0 Solving for it, we have a = Eθ|X [θ] Additionally ϕ (a) < 0 then a∗ = Eθ|X [θ] is a Bayesian Action. 99 / 117

204. Given X ∈ {Pθ, θ ∈ Θ} A family which is indexed by a parameter (random variable) θ Here, we change our Bayesian hat to the frequentist one This allows to make inferences about θ A solution is a decision procedure (decision rule) δ (x), that identiﬁes particular inference for each value of x that can be observed. 101 / 117

205. Given X ∈ {Pθ, θ ∈ Θ} A family which is indexed by a parameter (random variable) θ Here, we change our Bayesian hat to the frequentist one This allows to make inferences about θ A solution is a decision procedure (decision rule) δ (x), that identiﬁes particular inference for each value of x that can be observed. 101 / 117

206. A be the class of all possible realizations of δ(x), i.e. actions The Loss function L (θ, a) maps Θ × A −→ R Deﬁning a cost to the statistician when he takes the action a and the true value of the parameter is θ. Then we can deﬁne a decision function called Risk R (θ, δ) = EX|θ [L (θ|δ (X))] = X L (θ|δ (X)) f (x|θ) dx A frequentist risk on the performance of δ. 102 / 117

207. A be the class of all possible realizations of δ(x), i.e. actions The Loss function L (θ, a) maps Θ × A −→ R Deﬁning a cost to the statistician when he takes the action a and the true value of the parameter is θ. Then we can deﬁne a decision function called Risk R (θ, δ) = EX|θ [L (θ|δ (X))] = X L (θ|δ (X)) f (x|θ) dx A frequentist risk on the performance of δ. 102 / 117

208. Therefore Since the risk function is deﬁned as an average loss with respect to a sample space it is called the frequentist risk. Let D be the collection of all measurable decision rules There are several ways for assigning the preference among the rules in D. 103 / 117

209. Therefore Since the risk function is deﬁned as an average loss with respect to a sample space it is called the frequentist risk. Let D be the collection of all measurable decision rules There are several ways for assigning the preference among the rules in D. 103 / 117

210. Furthermore Some of them are The Minimax Principle Γ-minimax Principle Minimax Principle etc The one we are interested is The Bayes principle 104 / 117

215. Under the Bayes principle Bayes risk r (π, δ) = R (θ, δ) π (dθ) = EθR (θ, δ) Where there is a δπ , called Bayes rule, minimizing the risk δπ = arg inf δ∈D r (π, δ) Bayes risk of the prior distribution π (Bayes envelope function) is r (π) = r (π, δπ) 105 / 117

218. Bayes Envelope Function Definition Definition The Bayes Envelope is the maximal reward rate a player could achieve had he known in advance the relative frequencies of the other players. In particular, we define the following function as r (π, δ) = Eθ EX|θ [L (θ, δ (X))] Therefore the Bayes action as Bayes Rules looks like δ∗ (x) = arg min δ∈D r (π, δ) 106 / 117

221. Actually A classic Bayes Rule The Naive Bayes Rules for classiﬁcation using Gaussian’s for classiﬁcation R1 : P (ω1|x) > P (ω2|x) R2 : P (ω2|x) > P (ω1|x) 107 / 117

223. The Fubini’s Theorem (Informal Version) Theorem Suppose X and Y are σ-ﬁnite measure spaces, and suppose that X × Y is given the product measure: (µ × ν) (E) = inf    ∞ j=1 µ (Aj) ν (Bj) |E ⊂ ∪∞ j=1Aj × Bj    With any non-negative µ × ν-measurable function f, then X×Y f (x, y) d (µ × ν) (x, y) = Y X f (x, y) dµ (x) dν (y) 109 / 117

224. The Fubini’s Theorem (Informal Version) Theorem Suppose X and Y are σ-ﬁnite measure spaces, and suppose that X × Y is given the product measure: (µ × ν) (E) = inf    ∞ j=1 µ (Aj) ν (Bj) |E ⊂ ∪∞ j=1Aj × Bj    With any non-negative µ × ν-measurable function f, then X×Y f (x, y) d (µ × ν) (x, y) = Y X f (x, y) dµ (x) dν (y) 109 / 117

225. Implications with the Expected Value We have by the Fubini’s Theorem r (π, δ) = Eθ EX|θ [L (θ, δ (X))] = EX Eθ|X [L (θ, δ (X))] Where the posterior expected loss ρ (π, δ) = Eθ|X [L (θ, δ (X))] 110 / 117

226. Implications with the Expected Value We have by the Fubini’s Theorem r (π, δ) = Eθ EX|θ [L (θ, δ (X))] = EX Eθ|X [L (θ, δ (X))] Where the posterior expected loss ρ (π, δ) = Eθ|X [L (θ, δ (X))] 110 / 117

227. Therefore r (π, δ) is minimized for any ﬁxed x When ρ (π, δ) is minimized, for any ﬁxed x, δB (x) = a∗ (x) where * represent the optimal action. Basically This result links the conditional Bayesian and decision theoretic frequentist inference: The frequentist Bayes rule conditional on X is the Bayes action. 111 / 117

228. Therefore r (π, δ) is minimized for any ﬁxed x When ρ (π, δ) is minimized, for any ﬁxed x, δB (x) = a∗ (x) where * represent the optimal action. Basically This result links the conditional Bayesian and decision theoretic frequentist inference: The frequentist Bayes rule conditional on X is the Bayes action. 111 / 117

229. What happens when we have the Squared Loss? The Bayes rule is the posterior expectation δB (x) = Θ θf (x|θ) π (θ) dθ Θ f (x|θ) π (θ) dθ Not only that, in the case of L (θ, a) = w (θ) (θ − a)2 112 / 117

230. What happens when we have the Squared Loss? The Bayes rule is the posterior expectation δB (x) = Θ θf (x|θ) π (θ) dθ Θ f (x|θ) π (θ) dθ Not only that, in the case of L (θ, a) = w (θ) (θ − a)2 112 / 117

231. We have The following Bayes Rule δB (x) = Θ w (θ) θf (x|θ) π (θ) dθ Θ w (θ) f (x|θ) π (θ) dθ 113 / 117

232. Furthermore According to a Bayes principle A rule δ1 (X) is preferred to δ2 (X) if r (π, δ1) < r (π, δ2) The frequentists use Bayes principle to compare frequentist risks of the rules R (θ, δ1) and R (θ, δ2). 114 / 117

233. Furthermore According to a Bayes principle A rule δ1 (X) is preferred to δ2 (X) if r (π, δ1) < r (π, δ2) The frequentists use Bayes principle to compare frequentist risks of the rules R (θ, δ1) and R (θ, δ2). 114 / 117

234. Analysis of frequentist risk It leads to various concepts as 1 minimaxity, 2 admissibility, 3 unbiasedness, 4 equivariance, 5 etc. 115 / 117

235. D. V. Lindley and L. D. Phillips, “Inference for a bernoulli process (a bayesian view),” The American Statistician, vol. 30, no. 3, pp. 112–119, 1976. C. Aschwanden, “Not even scientists can easily explain p-values.” https://ﬁvethirtyeight.com/features/not-even-scientists-can-easily- explain-p-values. Online; accessed 24 Noviembre 2015. K. P. F.R.S., “X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 50, no. 302, pp. 157–175, 1900. 116 / 117

236. P. Bickel and K. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics. No. v. 1 in Mathematical Statistics: Basic Ideas and Selected Topics, Prentice Hall, 2001. R. A. Fisher, “On the mathematical foundations of theoretical statistics,” Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, vol. 222, no. 594-604, pp. 309–368, 1922. G. Casella and R. L. Berger, Statistical inference, vol. 2. Duxbury Paciﬁc Grove, CA, 2002. S. M. Kay, Fundamentals of statistical signal processing. Prentice Hall PTR, 1993. 117 / 117

2.03 bayesian estimation

More Related Content

What's hot (20)

Similar to 2.03 bayesian estimation (20)

More from Andres Mendez-Vazquez (20)

Recently uploaded (20)

2.03 bayesian estimation