Consider $n$ i.i.d observations from a normal distribution with unknown mean, $\mu$, and unknown variance $\sigma^2$, ie, $y_i \sim i.i.d \ N(\mu, \sigma^2)$ for $i = 1, 2, \cdots, n$.

Let $\boldsymbol{\theta} = (\mu, \sigma)'$ be the parameterisation for the two unknown parameters. Using Jeffrey's Prior and assuming a priori independence between $\mu$ and $\sigma$, we can show that a noninformative prior for $\boldsymbol{\theta}$ can be written as $p(\mu, \sigma) \propto \frac{1}{\sigma}$. The posterior is thus given by: \begin{align*} p(\mu, \sigma|\mathbf{y}) & \propto L(\mu, \sigma|\mathbf{y})p(\mu,\sigma) \ \ \text{where} \ \ L \ \ \text{denotes the likelihood function}. \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

I want to show that this prior is invariant to the parameterisation where we define $\boldsymbol{\eta} = (\mu, \psi)'$ where $\psi = \ln(\sigma)$. The following is my working, please advise if my arguments and reasoning are correct.

Given $\boldsymbol{\theta} = (\mu, \sigma)$ and $\boldsymbol{\eta} = (\mu, \psi)$ where $\psi = \ln(\sigma) \implies \exp(\psi) = \sigma$, to show that invariance holds, we need to show that $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$ and $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$ provide equivalent expressions. We begin by finding an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$.

Since, $$p(\mathbf{y}|\boldsymbol{\theta}) = L(\boldsymbol{\theta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\sigma^2\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - \mu)^2\right]$$ Then, $$L(\boldsymbol{\eta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\exp(2\psi)\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$$ The log-likelihood is thus, $$l(\boldsymbol{\eta}|\mathbf{y}) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\left(2\psi\right) - \frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2$$ Assuming a priori independence between $\mu$ and $\psi$ implies $p(\mu, \psi) = p(\mu)p(\psi)$, and applying Jeffreys' Prior on each parameter yields, \begin{align*}p(\mu) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \mu^2}\right]\right|^{\frac{1}{2}} \\ p(\psi) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \psi^2}\right]\right|^{\frac{1}{2}}\end{align*}

Since $\frac{\partial^2 l}{\partial \mu^2} = -\frac{n}{\exp(2\psi)}$ and $\frac{\partial^2 l}{\partial \psi^2} = -\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2$, we have: \begin{align*} p(\mu) \propto \left|-E\left[-\frac{n}{\exp(2\psi)}\right]\right|^{\frac{1}{2}} = \left(\frac{n}{\exp(2\psi)}\right)^{\frac{1}{2}} \propto c \ \ \ \text{where} \ c \ \text{is a constant} \end{align*} $$p(\psi) \propto \left|-E\left[-\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2\right]\right|^{\frac{1}{2}} =\left(\frac{2}{\exp(2\psi)}E\left[\sum_{i=1}^n (y_i-\mu)^2\right]\right)^{\frac{1}{2}} = \left|\frac{2n\exp(2\psi)}{\exp(2\psi)} \right|^{\frac{1}{2}} \propto c$$

Hence $p(\boldsymbol{\eta}) \propto c$, and $p(\boldsymbol{\eta}|\mathbf{y}) \propto \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$. Next we find an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$, assuming we use the correct, non-informative prior for $\boldsymbol{\theta}$, ie, $p(\boldsymbol{\theta}) \propto \frac{1}{\sigma}$. Computing the Jacobian is as follows: \begin{align*} \displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|} = \left|\left[\begin{matrix} \mu \\ \sigma \end{matrix}\right]\left[\begin{matrix} \frac{\partial}{\partial \mu} & \frac{\partial}{\partial \psi} \end{matrix}\right] \right| = \left|\left[\begin{matrix} \frac{\partial \mu}{\partial \mu} & \frac{\partial \mu}{\partial \psi} \\ \frac{\partial \sigma}{\partial \mu} & \frac{\partial \sigma}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \frac{\partial\exp(\psi)}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \exp(\psi) \end{matrix}\right]\right| = \exp(\psi) \end{align*}

Thus we have, \begin{align*} p(\boldsymbol{\eta}|\mathbf{y}) & \propto \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n + \psi)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

Which is exactly the expression we obtained, hence the invariance property holds if $p(\mu, \sigma) \propto \frac{1}{\sigma}$.


To "Illustrate the invariance property of a noninformative prior" (assuming by noninformative you mean Jeffreys) just work with the general case of Jeffreys prior being the square root of the determinant of the fisher information and take the jacobian inside the square root. I would write it out, but wikipedia has already done it


that should be enough to show you that it is invariant under reparameterization.

  • $\begingroup$ Thanks, actually I'm just trying to show it for a specific case here, but the general proof is even better :) $\endgroup$
    – Trts
    Aug 11 '13 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.