I'm following these notes to compute the conjugate prior of a normal distribution with unknown mean and known variance. At some point they claim:

$p(D|\mu) \propto exp(-\frac{n}{2\sigma²}(\overline{x}-\mu)^2) \propto N(\overline{x},\theta,\frac{1}{n})$

but then they claim that the natural conjugate prior has the form:

$p(\mu) \propto exp(-\frac{n}{2\sigma_0^2}(\mu-\mu_0)^2) \propto N(\mu|\mu_0,\sigma_0^2)$

Is this mathematics?

Contrast this with these other notes that write the likelihood:

$L(\mu|x_1,\ldots,x_n) = c exp(-\frac{n(\mu-\overline{x}}{2\sigma^2}-\frac{\sum (x_i-\overline{x})^2}{2\sigma^2})$

then they claim that one should focus on the term $exp(-\frac{n(\mu-\overline{x}}{2\sigma^2})$ which is said to be clearly propotional to a normal distribution of $\mu$!!!

What is going on here? How can I compute the conjugate prior of a normal distribution with unknown mean?


A good account of this case can be found in De Groot's "Probability and Statistics"


Assume you model the likelihood through the sufficient statistic $\bar{X}$: \begin{equation} P(D|\mu) \propto \textrm{Exp}(-\frac{n}{2\sigma²}(\overline{x}-\mu)^2) \propto N(\overline{x},\theta,\frac{1}{n}) \end{equation} And use the following as a prior for $\mu$: \begin{equation} P(\mu) \propto \textrm{Exp}(-\frac{n}{2\sigma_0^2}(\mu-\mu_0)^2) \propto N(\mu|\mu_0,\sigma_0^2) \end{equation}

We need to show that the product of these two distributions has a kernel which is the kernel (as a function of $\mu$) of a normal distribution; so we can ignore all of the normalizing constants and any terms not containing $\mu$.

\begin{equation} \begin{split} P(D|\mu)P(\mu) & \propto \textrm{Exp}(-\frac{n}{2\sigma²}(\overline{x}-\mu)^2) \textrm{Exp}(-\frac{n}{2\sigma_0^2}(\mu-\mu_0)^2) \\ & = \textrm{Exp}(-\frac{n}{2}[\frac{1}{\sigma^2}(\bar{X}^2 - 2\bar{X}\mu + \mu^2)+\frac{1}{\sigma_0^2}(\mu^2 - 2\mu\mu_0 + \mu_0^2)]) \end{split} \end{equation}

Throwing away terms which do not contain $\mu$, we get: \begin{equation} \begin{split} \textrm{Exp}(-\frac{n}{2}[\frac{1}{\sigma^2}(2\bar{X}\mu + \mu^2)+\frac{1}{\sigma_0^2}(\mu^2 - 2\mu\mu_0)]) & \propto \textrm{Exp}(-\frac{n}{2}[\mu^2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2}) +\mu(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})]) \end{split} \end{equation}

Now to get a quadratic in $\mu$ (and thus an appropriately specified normal distribution) we complete the square:

\begin{equation} \begin{split} \textrm{Exp}(-\frac{n}{2}[\mu^2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2}) +\mu(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})]) & = \textrm{Exp}(-\frac{n}{2}[a(\mu-h)^2 + k]) \end{split} \end{equation}

For some $a$, $h$, and a term $k$ which does not involve $\mu$ and can be removed.

Specifically, ignoring $k$, we have that $a = (\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})$, and that $h = -\frac{(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})}{2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})}$.

This gives us our mean ($h$), and the reciprocal of our variance (up to rescaling by the $n$ factor present):

\begin{equation} \begin{split} \textrm{Exp}(-\frac{n}{2}[\mu^2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2}) +\mu(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})]) & \propto \textrm{Exp}(-\frac{n(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})}{2}[(\mu-\frac{(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})}{2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})})^2]) \end{split} \end{equation}

Which is a $\textrm{N}(\frac{(\frac{2\bar{X}}{\sigma^2}-\frac{2\mu_0}{\sigma_0^2})}{2(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})}, (\frac{1}{n}(\frac{1}{\sigma^2} + \frac{1}{\sigma_0^2})^{-1})$.

Which corresponds to the result given here up to some canceling of values. The conjugate prior specified in the problem also has the $n$ factor, which the Wikipedia page doesn't have. You can just collapse that into $\sigma_0$, though, to get the same result they have.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.