Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Consider $n$ i.i.d observations from a normal distribution with unknown mean, $\mu$, and unknown variance $\sigma^2$, ie, $y_i \sim i.i.d \ N(\mu, \sigma^2)$ for $i = 1, 2, \cdots, n$.

Let $\boldsymbol{\theta} = (\mu, \sigma)'$ be the parameterisation for the two unknown parameters. Using Jeffrey's Prior and assuming a priori independence between $\mu$ and $\sigma$, we can show that a noninformative prior for $\boldsymbol{\theta}$ can be written as $p(\mu, \sigma) \propto \frac{1}{\sigma}$. The posterior is thus given by: \begin{align*} p(\mu, \sigma|\mathbf{y}) & \propto L(\mu, \sigma|\mathbf{y})p(\mu,\sigma) \ \ \text{where} \ \ L \ \ \text{denotes the likelihood function}. \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

I want to show that this prior is invariant to the parameterisation where we define $\boldsymbol{\eta} = (\mu, \psi)'$ where $\psi = \ln(\sigma)$. The following is my working, please advise if my arguments and reasoning are correct.

Given $\boldsymbol{\theta} = (\mu, \sigma)$ and $\boldsymbol{\eta} = (\mu, \psi)$ where $\psi = \ln(\sigma) \implies \exp(\psi) = \sigma$, to show that invariance holds, we need to show that $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$ and $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$ provide equivalent expressions. We begin by finding an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$.

Since, $$p(\mathbf{y}|\boldsymbol{\theta}) = L(\boldsymbol{\theta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\sigma^2\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - \mu)^2\right]$$ Then, $$L(\boldsymbol{\eta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\exp(2\psi)\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$$ The log-likelihood is thus, $$l(\boldsymbol{\eta}|\mathbf{y}) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\left(2\psi\right) - \frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2$$ Assuming a priori independence between $\mu$ and $\psi$ implies $p(\mu, \psi) = p(\mu)p(\psi)$, and applying Jeffreys' Prior on each parameter yields, \begin{align*}p(\mu) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \mu^2}\right]\right|^{\frac{1}{2}} \\ p(\psi) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \psi^2}\right]\right|^{\frac{1}{2}}\end{align*}

Since $\frac{\partial^2 l}{\partial \mu^2} = -\frac{n}{\exp(2\psi)}$ and $\frac{\partial^2 l}{\partial \psi^2} = -\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2$, we have: \begin{align*} p(\mu) \propto \left|-E\left[-\frac{n}{\exp(2\psi)}\right]\right|^{\frac{1}{2}} = \left(\frac{n}{\exp(2\psi)}\right)^{\frac{1}{2}} \propto c \ \ \ \text{where} \ c \ \text{is a constant} \end{align*} $$p(\psi) \propto \left|-E\left[-\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2\right]\right|^{\frac{1}{2}} =\left(\frac{2}{\exp(2\psi)}E\left[\sum_{i=1}^n (y_i-\mu)^2\right]\right)^{\frac{1}{2}} = \left|\frac{2n\exp(2\psi)}{\exp(2\psi)} \right|^{\frac{1}{2}} \propto c$$

Hence $p(\boldsymbol{\eta}) \propto c$, and $p(\boldsymbol{\eta}|\mathbf{y}) \propto \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$. Next we find an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$, assuming we use the correct, non-informative prior for $\boldsymbol{\theta}$, ie, $p(\boldsymbol{\theta}) \propto \frac{1}{\sigma}$. Computing the Jacobian is as follows: \begin{align*} \displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|} = \left|\left[\begin{matrix} \mu \\ \sigma \end{matrix}\right]\left[\begin{matrix} \frac{\partial}{\partial \mu} & \frac{\partial}{\partial \psi} \end{matrix}\right] \right| = \left|\left[\begin{matrix} \frac{\partial \mu}{\partial \mu} & \frac{\partial \mu}{\partial \psi} \\ \frac{\partial \sigma}{\partial \mu} & \frac{\partial \sigma}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \frac{\partial\exp(\psi)}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \exp(\psi) \end{matrix}\right]\right| = \exp(\psi) \end{align*}

Thus we have, \begin{align*} p(\boldsymbol{\eta}|\mathbf{y}) & \propto \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n + \psi)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

Which is exactly the expression we obtained, hence the invariance property holds if $p(\mu, \sigma) \propto \frac{1}{\sigma}$.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

To "Illustrate the invariance property of a noninformative prior" (assuming by noninformative you mean Jeffreys) just work with the general case of Jeffreys prior being the square root of the determinant of the fisher information and take the jacobian inside the square root. I would write it out, but wikipedia has already done it

http://en.wikipedia.org/wiki/Jeffreys_prior

that should be enough to show you that it is invariant under reparameterization.

share|improve this answer
    
Thanks, actually I'm just trying to show it for a specific case here, but the general proof is even better :) –  TrueTears Aug 11 '13 at 6:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.