# Bayesian inference exercise

I am learning online Bayesian Statistics and I have a test in a couple of days. I have no idea how to solve this exercise, any help will be appreciated. There might be something similar in the quiz...

Statistical decision theory: a decision-theoretic approach to the estimation of an unknown parameter $\theta$ introduces the loss function $L(\theta, a)$ which, loosely speaking, gives the cost of deciding that the parameter has the value $a$, when it is in fact equal to $\theta$. The estimate $a$ can be chosen to minimize the posterior expected loss, $$E(L(a|y))= \int L(\theta,a)p(\theta|y)d\theta$$ This optimal choice of $a$ is called a Bayes estimate for the loss function $L$. Show that:

(a) If $L(\theta, a) = (\theta − a)^2$ (squared error loss), then the posterior mean, $E(\theta|y)$, if it exists, is the unique Bayes estimate of $\theta$.

(b) If $L(\theta, a) = |\theta − a|$, then any posterior median of $\theta$ is a Bayes estimate of $\theta$.

(c) If $k_0$ and $k_1$ are non negative numbers, not both zero, and $L(\theta,a)= k_0(\theta−a)$ if $\theta\geq a$, $k_1(a−\theta)$ if $\theta<a$, then any $k_0$ quantile of the posterior distribution $p(\theta|y)$ is a Bayes estimate of $\theta$.

Here is the derivation for part (a) using slightly different notation.

If $L(\theta, \hat \theta) = (\theta - \hat \theta)^2$ then we minimize the posterior risk for a given (fixed) value of $z$ as:

\begin{aligned} \arg \min_{\hat \theta} r(\hat \theta | z) & = \arg \min_{\hat \theta} \int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta \\ \frac{\partial}{\partial \hat \theta} & = 2 \int (\hat \theta - \theta) \pi(\theta | z) \ d\theta = 0 \\ \hat \theta \int \pi(\theta | z) \ d\theta & = \int \theta \pi(\theta | z) \ d\theta \\ \hat \theta & = \int \theta \pi(\theta | z) \ d\theta = E(\theta | z) \end{aligned}

For the last step recall that $\int \pi(\theta | z) \ d\theta = 1$ since this is the integral from $- \infty$ to $\infty$ of a probability distribution function, which must be equal to 1.

Alternatively, we can prove this using a different approach. Since the integral $\int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta$ is over $\theta$ it is effectively the same as the expectation over $\theta$ when treating $\hat \theta$ as a constant.

$$\int (\theta - \hat \theta)^2 \pi(\theta | z) \ d\theta = E_\theta[ (\theta - \hat \theta)^2 | z]$$

using expectation algebra this can be re-written in a different form (dropping the $|z$ for notational convenience):

\begin{aligned} E_\theta[ (\theta - \hat \theta)^2] & = E_\theta[ \theta^2 - 2 \theta \hat \theta + \hat \theta^2 ] \\ & = E_\theta[\theta^2] - 2 \hat \theta E_\theta[\theta] + \hat \theta^2 \end{aligned}

We can then differentiate w.r.t. $\hat \theta$ giving:

\begin{aligned} \frac{\partial}{\partial \hat \theta} E_\theta[ (\theta - \hat \theta)^2] & = \frac{\partial}{\partial \hat \theta} E_\theta[\theta^2] - 2 \hat \theta E_\theta[\theta] + \hat \theta^2 \\ & = - 2 E_\theta[\theta] + 2 \hat \theta = 0 \\ \hat \theta &= E_\theta[\theta] \end{aligned}