I am learning online Bayesian Statistics and I have a test in a couple of days. I have no idea how to solve this exercise, any help will be appreciated. There might be something similar in the quiz...

Statistical decision theory: a decision-theoretic approach to the estimation of an unknown parameter $\theta$ introduces the loss function $L(\theta, a)$ which, loosely speaking, gives the cost of deciding that the parameter has the value $a$, when it is in fact equal to $\theta$. The estimate $a$ can be chosen to minimize the posterior expected loss, $$E(L(a|y))= \int L(\theta,a)p(\theta|y)d\theta$$ This optimal choice of $a$ is called a Bayes estimate for the loss function $L$. Show that:

(a) If $L(\theta, a) = (\theta − a)^2$ (squared error loss), then the posterior mean, $E(\theta|y)$, if it exists, is the unique Bayes estimate of $\theta$.

(b) If $L(\theta, a) = |\theta − a|$, then any posterior median of $\theta$ is a Bayes estimate of $\theta$.

(c) If $k_0$ and $k_1$ are non negative numbers, not both zero, and $L(\theta,a)= k_0(\theta−a)$ if $\theta\geq a$, $k_1(a−\theta)$ if $\theta<a$, then any $k_0$ quantile of the posterior distribution $p(\theta|y)$ is a Bayes estimate of $\theta$.


Here is the derivation for part (a) using slightly different notation.

If $L(\theta, \hat \theta) = (\theta - \hat \theta)^2$ then we minimize the posterior risk for a given (fixed) value of $z$ as:

\begin{aligned} \arg \min_{\hat \theta} r(\hat \theta | z) & = \arg \min_{\hat \theta} \int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta \\ \frac{\partial}{\partial \hat \theta} & = 2 \int (\hat \theta - \theta) \pi(\theta | z) \ d\theta = 0 \\ \hat \theta \int \pi(\theta | z) \ d\theta & = \int \theta \pi(\theta | z) \ d\theta \\ \hat \theta & = \int \theta \pi(\theta | z) \ d\theta = E(\theta | z) \end{aligned}

For the last step recall that $ \int \pi(\theta | z) \ d\theta = 1$ since this is the integral from $- \infty$ to $\infty$ of a probability distribution function, which must be equal to 1.

Alternatively, we can prove this using a different approach. Since the integral $ \int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta $ is over $\theta$ it is effectively the same as the expectation over $\theta$ when treating $\hat \theta$ as a constant.

$$ \int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta = E_\theta[ (\theta - \hat \theta)^2 | z]$$

using expectation algebra this can be re-written in a different form (dropping the $|z$ for notational convenience):

\begin{aligned} E_\theta[ (\theta - \hat \theta)^2] & = E_\theta[ \theta^2 - 2 \theta \hat \theta + \hat \theta^2 ] \\ & = E_\theta[\theta^2] - 2 \hat \theta E_\theta[\theta] + \hat \theta^2 \end{aligned}

We can then differentiate w.r.t. $\hat \theta$ giving:

\begin{aligned} \frac{\partial}{\partial \hat \theta} E_\theta[ (\theta - \hat \theta)^2] & = \frac{\partial}{\partial \hat \theta} E_\theta[\theta^2] - 2 \hat \theta E_\theta[\theta] + \hat \theta^2 \\ & = - 2 E_\theta[\theta] + 2 \hat \theta = 0 \\ \hat \theta &= E_\theta[\theta] \end{aligned}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.