Suppose that $X_1,\ldots,X_n\mid\Theta = \theta\stackrel{iid}{\sim}N(\theta, \sigma^2)$ and $\Theta\sim N(\mu_0, \sigma_0^2)$ where we assume $\sigma^2$ to be known. The parameter $\mu_0$ and $\sigma_0^2$ are also assumed to be known and are part of the prior specification. If we set $X = (X_1,\ldots,X_n)$, then $$\Theta\mid X \sim N(\mu_1, \sigma_1^2)$$ with $\mu_1 = \sigma_1^2\bigg(\dfrac{\mu_0}{\sigma_0^2} + \dfrac{n\overline{X_n}}{\sigma^2}\bigg)$ and $\dfrac{1}{\sigma_1^2} = \dfrac{1}{\sigma_0^2} + \dfrac{1}{\sigma^2}.$

Exercise: Verify the above calculations.

What I've tried: I know that $f_{\Theta\mid X}(\theta\mid x) \propto f_{X\mid\Theta}(x\mid\theta)\, f_\Theta(\theta)$. Hence, $$f_{\Theta\mid X}(\theta\mid x)\propto \prod_{i = 1}^n\dfrac{1}{\sqrt{2\pi\sigma^2}}\exp{\bigg(-\dfrac{(x_i-\theta)^2}{2\sigma^2}\bigg)}\dfrac{1}{\sqrt{2\pi\sigma_0^2}}\exp\bigg(-\dfrac{(\theta-\mu_0)^2}{2\sigma_0^2}\bigg).$$ I tried to look at this product and find the matching distribution by just looking at the exponential and the rest of the function separately. If we fast forward through some calculations, and just look at the exponential we get: $\exp\bigg[-\bigg(\dfrac{\sum_{i = 1}^nx_i^2 - 2\theta\sum_{i =1}^n x_i + n\theta^2}{2\sigma^2} + \dfrac{n\theta^2 - 2n\theta\mu_0 + n\mu_0^2}{2\sigma_0^2}\bigg)\bigg]$. I can see that the $\sum_{i = 1}^nx_i$ part can be written as $n\overline{X_n}$, but don't know how to make the other terms cancel out.

I looked at the other part of the function as well: $$\bigg(\dfrac{1}{\sqrt{2\pi\sigma^2}}\bigg)^n\dfrac{1}{\sqrt{2\pi\sigma_0^2}},$$ but again I'm not sure how this translates to $1/\sigma_1^2 = 1/\sigma_0^2 + 1/\sigma^2$.

Question: How do I solve this exercise and show that $\Theta\mid X \sim N(\mu_1,\sigma_1^2)$?

Thanks in advance!

  • 1
    $\begingroup$ See also here: en.wikipedia.org/wiki/… $\endgroup$ – nicola Dec 30 '17 at 15:27
  • 1
    $\begingroup$ One thing you need to bear in mind is that you have chosen $\theta$ as your argument in your posterior pdf. So when you try to show the result, you should try to complete the square in terms of $\theta$ inside the exponent, but not the other stuff. $\endgroup$ – BGM Dec 30 '17 at 16:07
  • $\begingroup$ @BGM Thanks for your comments guys! Unfortunately I'm not able to solve this exercise by myself. Could you help me out a bit more? $\endgroup$ – titusAdam Jan 23 '18 at 13:47
  • $\begingroup$ It's apparently not that easy to solve this exercise.. $\endgroup$ – titusAdam Jan 23 '18 at 14:29
  • $\begingroup$ As said in above, note that the exponent can be viewed as a quadratic expression in $\theta$. Collect the $\theta^2$ and $\theta$ term, then complete the square in $\theta$ according to the coefficients of these two terms, and see what you obtained at last. $\endgroup$ – BGM Jan 23 '18 at 16:09

As you started, from the Bayes rule we know that $\ f_{\Theta \ \vert \ X} (\theta \ \vert \ x) \propto f_{X \ \vert \ \Theta} (x \ \vert \ \theta) \ f_{\Theta}(\theta)$. On the right hand side, the first density is the Likelihood of $X$ given $\theta$ and the second density is our prior.

Hence, given that $X=(X_1, X_2,\ldots, X_n)$ and $X_1,..., X_n \ \vert \ \Theta = \theta \ $ ~ $N(\theta, \lambda^{-1})$, where $\lambda = \frac{1}{\sigma^2}$ is the precision, we can calculate the likelihood. In the calculations we will drop all the terms, which are not dependent on $\theta$.

\begin{align} f_{X \ \vert \ \Theta} (x \ \vert \ \theta) & = \prod\limits_{k=1}^{n}f_{X_k \ \vert \ \Theta} (x_k \ \vert \ \theta) = (2\pi\lambda^{-1})^{\frac{n}{2}} \ e^{-\sum_{k=1}^n\frac{\lambda}{2}(x_k-\theta)^2} \\ & \propto e^{-\frac{\lambda}{2}\sum_{k=1}^n(x_k-\theta)^2} = e^{-\frac{\lambda}{2}\sum\limits_{k=1}^n(x_k - \overline{x} + \overline{x} -\theta)^2} \\ & = e^{-\frac{\lambda}{2}\sum\limits_{k=1}^n(x_k - \overline{x})^2 + (\overline{x} -\theta)^2 + 2(x_k - \overline{x})(\overline{x} -\theta)} \\ & \propto e^{-\frac{n\lambda}{2}(\overline{x} -\theta)^2 - \frac{\lambda}{2}(\overline{x}-\theta)\sum\limits_{k=1}^n(x_k-\overline{x})} \\ & = e^{-\frac{n\lambda}{2}(\overline{x} -\theta)^2} \end{align}

where $\overline{x} = \frac{1}{n}\sum\limits_{k=1}^nx_k$ is the mean of the observations. Now, we know that $f_{\Theta}(\theta) \ $~ $N(\mu_0, \lambda_0)$, where $\lambda_0=\frac{1}{\sigma_0^2}$ is the prior precision. Hence, we can write down the RHS of our Bayes identity as:

$f_{X \ \vert \ \Theta} (x \ \vert \ \theta) \ f_{\Theta}(\theta) \propto e^{-\frac{n\lambda}{2}(\overline{x} -\theta)^2} \ (2\pi \lambda_0^{-1})^{\frac{1}{2}}e^{-\frac{\lambda_0}{2}(\theta - \mu_0)^2} \propto e^{-\left(\frac{\lambda_0}{2}(\theta - \mu_0)^2+\frac{n\lambda}{2}(\overline{x} -\theta)^2\right)}$

We would like to transform the resulting expression into the kernel of a normal distribution. For clarity, I will just work on the exponent. A usual way to solve this would be to separate $\theta$ from everything else.

$-\left(\frac{\lambda_0}{2}(\theta - \mu_0)^2+\frac{n\lambda}{2}(\overline{x} -\theta)^2\right) = -\left(\frac{\lambda_0}{2}(\theta^2+\mu_0^2-2\theta\mu_0) + \frac{n\lambda}{2}(\overline{x}^2 + \theta^2 - 2\theta\overline{x})\right) = -\left(\frac{\lambda_0 + n\lambda}{2}(\theta^2 - 2\frac{\lambda_0\mu_0 + \overline{x}n\lambda}{\lambda_0+n\lambda}\theta) + \frac{\lambda_0\mu_0^2+ n\lambda\overline{x}^2}{2}\right) \propto -\frac{\lambda_0 + n\lambda}{2}\left(\theta - \frac{\lambda_0\mu_0 + \overline{x}n\lambda}{\lambda_0+n\lambda}\right)^2$

In the last proportion we completed the square. This is the kernel of a normal distribution with the following parameters:

$\mu_1 = \frac{\lambda_0\mu_0 + \overline{x}n\lambda}{\lambda_0+n\lambda} = \frac{\lambda_0}{\lambda_0+n\lambda}\mu_0 + \frac{n\lambda}{\lambda_0+n\lambda}\overline{x}$, the posterior mean is a weighted average of the measurement and the prior means.

$\lambda_1 = \lambda_0 + n\lambda$, the posterior precision is a linear function of the number of observations, their precision, and the prior precision.

Substituting variances in the place of precisions yields your results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.