# Illustrate the invariance property of a noninformative prior

Consider $n$ i.i.d observations from a normal distribution with unknown mean, $\mu$, and unknown variance $\sigma^2$, ie, $y_i \sim i.i.d \ N(\mu, \sigma^2)$ for $i = 1, 2, \cdots, n$.

Let $\boldsymbol{\theta} = (\mu, \sigma)'$ be the parameterisation for the two unknown parameters. Using Jeffrey's Prior and assuming a priori independence between $\mu$ and $\sigma$, we can show that a noninformative prior for $\boldsymbol{\theta}$ can be written as $p(\mu, \sigma) \propto \frac{1}{\sigma}$. The posterior is thus given by: \begin{align*} p(\mu, \sigma|\mathbf{y}) & \propto L(\mu, \sigma|\mathbf{y})p(\mu,\sigma) \ \ \text{where} \ \ L \ \ \text{denotes the likelihood function}. \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

I want to show that this prior is invariant to the parameterisation where we define $\boldsymbol{\eta} = (\mu, \psi)'$ where $\psi = \ln(\sigma)$. The following is my working, please advise if my arguments and reasoning are correct.

Given $\boldsymbol{\theta} = (\mu, \sigma)$ and $\boldsymbol{\eta} = (\mu, \psi)$ where $\psi = \ln(\sigma) \implies \exp(\psi) = \sigma$, to show that invariance holds, we need to show that $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$ and $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$ provide equivalent expressions. We begin by finding an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$.

Since, $$p(\mathbf{y}|\boldsymbol{\theta}) = L(\boldsymbol{\theta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\sigma^2\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - \mu)^2\right]$$ Then, $$L(\boldsymbol{\eta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\exp(2\psi)\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$$ The log-likelihood is thus, $$l(\boldsymbol{\eta}|\mathbf{y}) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\left(2\psi\right) - \frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2$$ Assuming a priori independence between $\mu$ and $\psi$ implies $p(\mu, \psi) = p(\mu)p(\psi)$, and applying Jeffreys' Prior on each parameter yields, \begin{align*}p(\mu) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \mu^2}\right]\right|^{\frac{1}{2}} \\ p(\psi) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \psi^2}\right]\right|^{\frac{1}{2}}\end{align*}

Since $\frac{\partial^2 l}{\partial \mu^2} = -\frac{n}{\exp(2\psi)}$ and $\frac{\partial^2 l}{\partial \psi^2} = -\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2$, we have: \begin{align*} p(\mu) \propto \left|-E\left[-\frac{n}{\exp(2\psi)}\right]\right|^{\frac{1}{2}} = \left(\frac{n}{\exp(2\psi)}\right)^{\frac{1}{2}} \propto c \ \ \ \text{where} \ c \ \text{is a constant} \end{align*} $$p(\psi) \propto \left|-E\left[-\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2\right]\right|^{\frac{1}{2}} =\left(\frac{2}{\exp(2\psi)}E\left[\sum_{i=1}^n (y_i-\mu)^2\right]\right)^{\frac{1}{2}} = \left|\frac{2n\exp(2\psi)}{\exp(2\psi)} \right|^{\frac{1}{2}} \propto c$$

Hence $p(\boldsymbol{\eta}) \propto c$, and $p(\boldsymbol{\eta}|\mathbf{y}) \propto \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$. Next we find an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$, assuming we use the correct, non-informative prior for $\boldsymbol{\theta}$, ie, $p(\boldsymbol{\theta}) \propto \frac{1}{\sigma}$. Computing the Jacobian is as follows: \begin{align*} \displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|} = \left|\left[\begin{matrix} \mu \\ \sigma \end{matrix}\right]\left[\begin{matrix} \frac{\partial}{\partial \mu} & \frac{\partial}{\partial \psi} \end{matrix}\right] \right| = \left|\left[\begin{matrix} \frac{\partial \mu}{\partial \mu} & \frac{\partial \mu}{\partial \psi} \\ \frac{\partial \sigma}{\partial \mu} & \frac{\partial \sigma}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \frac{\partial\exp(\psi)}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \exp(\psi) \end{matrix}\right]\right| = \exp(\psi) \end{align*}

Thus we have, \begin{align*} p(\boldsymbol{\eta}|\mathbf{y}) & \propto \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n + \psi)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

Which is exactly the expression we obtained, hence the invariance property holds if $p(\mu, \sigma) \propto \frac{1}{\sigma}$.

-