Conjugate Priors?

Question

Consider the following:$$$$ If $S^2$ is the sample variance based on a sample of size n from a normal population, we know that $\displaystyle\frac{(n-1)S^2}{\sigma ^2} \sim \chi ^2_{n-1}$. The conjugate prior for $\sigma ^2$ is the inverted gamma pdf $IG(\alpha,\beta)$, that is: $$ \displaystyle \pi(\sigma^2)=\frac{1}{\Gamma(\alpha)\beta^\alpha(\sigma^2)^{\alpha+1}}e^{-\frac{1}{\beta \sigma^2}} $$ I would like to show that the posterior distribution of $\sigma ^2$ is $\displaystyle IG(\alpha+\frac{n-1}{2},\Big[\frac{(n-1)S^2}{2}+\frac{1}{\beta}\Big]^{-1})$

$$$$

What exactly is a prior distribution, posterior distribution, and conjugate prior? If you can, please state both a formal and informal definition/description.

Answer 1

If you have a likelihood with a specified density $f(x|\theta)$, and you put a distribution on $\theta$ with density $\pi(\theta)$ (the prior), you can get a distribution on $\theta$ conditional on the observed data using Bayes rule.

\begin{equation} \pi(\theta|x) = \frac{f(x|\theta)\pi(\theta)}{f(x)} \end{equation}

This is the posterior distribution, the distribution of the parameters conditional on the data. Note that the prior represents what you think the distribution of the parameter should be a priori, and the posterior represents what you think the parameter should be once you observe data (taking into account your prior beliefs).

A conjugate prior is one in which the form of the posterior is in the same family of distributions as the prior; with adjusted parameters. This makes computing the posterior distribution much easier.

We have that if $\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}$ then $\frac{(n-1)S^2}{2} \sim \textrm{Gamma}(\frac{n-1}{2},\sigma^2)$.

Using the conjugacy of the Gamma to the second parameter of a Gamma distribution, we place a $\sigma^2 \sim \textrm{Gamma}(\alpha,\beta)$ prior on $\sigma^2$.

Which gives us $\sigma^2 | S^2 \sim \textrm{Gamma}(\alpha + \frac{n-1}{2},\beta + \frac{(n-1)S^2}{2})$

You can see the formula for how I got this here.