I have problems calculating the posterior mode and posterior mean of a posterior density. For example the following exercise:

The random variable $X$ has distribution (for $\theta \in[0,1]$): $$P(X = 1) = \theta, P(X=2) = 1-\theta.$$ The random variables $X_1, X_2, X_3$ are i.i.d. with the same distribution as $X$. The following observations have been made: $x_1 = 1, x_2 = 2, x_3 =2.$
a) Choose a uniform prior density for $\Theta$ and give the posterior density of $\Theta$ based on the data.
b) Give the posterior mean and mode estimate for $\theta$.
c) Do the same as b), only with the following prior density: $f_{\Theta}(\theta) \propto \theta^3(1-\theta)^51_{[0,1]}(\theta)$.

My thoughts:
$f_{\Theta}(\theta) = 1$ for $\theta \in [0,1]$
$f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = P(X_1 = 1, X_2 = 2, X_3 = 3\mid \theta) = \theta(1-\theta)^2 $
So $f_{\Theta \mid X}(\theta \mid x_1, x_2, x_3) \propto f_{\Theta}(\theta)f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = \theta(1-\theta)^2$

Posterior mode:
$logf_{\Theta \mid X} = log(\theta(1-\theta)^2) = log(\theta) + 2log(1-\theta)$
$\frac{d}{d\theta}logf_{\Theta \mid X} = \frac{1}{\theta}-\frac{2}{1-\theta} = 0 \Longleftrightarrow \theta = \frac{1}{3}$
This answer is correct according to my solutions book.

Posterior mean:
I think I have to calculate $\int_0^1\theta f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta)d\theta$.
So $E(\Theta) = \int_0^1 \theta^2(1-\theta)^2d\theta = .. = \frac{1}{30}$
The answer should be $\frac{2}{5}$ though...

$f_{\Theta \mid X}(\theta \mid x_1, x_2, x_3) \propto f_{\Theta}(\theta)f_{X\mid \Theta}(x_1,x_2,x_3 \mid \theta) = \theta^3(1-\theta)^5\theta(1-\theta)^2 = \theta^4 (1-\theta)^7$
When I calculate the mode in the same way as b) I get $\frac{4}{11}$, though it should be $\frac{4}{13}$. As for the mean, I don't even know how to solve that integral...

Sorry for the long text. I would appreciate it a lot if someone could explain what I'm doing wrong and how it's supposed to be done.


So the disagreement between your answer and the solutions for the first choice of prior follow from the fact that you have, correctly, calculated the posterior only up to the normalising constant, so we need to find that $$ \int_0^1 \theta(1-\theta)^2d\theta=\frac{1}{12} $$ so that we now have $$ f_{\Theta}(\theta)=12\theta(1-\theta)^2, \qquad\theta \in [0,1]. $$ So for part $b)$ we now have, multiplying your result by the proper normalising constant, that the result is $12 \times \frac{1}{30} = \frac{2}{5}$.

Repeating for part (c) for the mode when you don't need to find the normalising constant it should just be a case of $$ 0 = \frac{\partial \log f_{\Theta|X}(\theta)}{\partial \theta} = \frac{4}{\theta}-\frac{7}{1-\theta} \implies \theta = \frac{4}{11}. $$ which agrees with your work.

To find the mean it helps to identify the posterior with a Beta distribution, that is $$ \begin{align*} \int_0^{1}\theta^{4}(1-\theta)^{7}d\theta&=B(5,8) \underbrace{\int_0^1 \frac{\theta^4(1-\theta)^7}{B(5,8)}d\theta}_{=1} \end{align*} $$ where $B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$ and so the appropriate form of the posterior is $$ \begin{align*} f_{\Theta|X}(\theta) =\frac{1}{B(5,8)}\theta^4(1-\theta)^7, \qquad \mathbf{E}_{\Theta | X}\left[ \Theta \right] = \frac{5}{5 + 8} = \frac{5}{13}. \end{align*} $$ So check that with your solutions and see if I have made any obvious mistakes? If you haven't already seen the Beta function then it would have been possible, but tedious, to get the result by expanding $(1-\theta)^7$ using the Binomial theorem.

  • $\begingroup$ First of all, thanks a lot for your answer. I think I get it now. So for finding the posterior mean I first need to calculate the normalising constant. For finding the mode I don't need to do that, I just say $f_{\Theta | X} (\theta) \propto f_{X|\Theta}(x)f_{\Theta}(\theta)$ and then find the maximum. Is this correct? Also I understand the part with the Beta distribution, thank you! $\endgroup$ – Violet Oct 5 '17 at 11:24
  • $\begingroup$ Yes that is exactly correct - finding the mode depends on the 'shape' of the posterior and if $f$ is the unormalised density and $c f$ is the normalised then the maximum of $f$ is the same as the maximum of $cf$ so you don't require normalisation to calculate the mode. On the otherhand the expectation requires integrating over the function so that in general $\int x c f(x) dx = c \int xf(x) dx \neq \int x f(x) dx$. $\endgroup$ – Nadiels Oct 5 '17 at 12:17
  • $\begingroup$ I understand everything now, thanks again :) $\endgroup$ – Violet Oct 5 '17 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.