How to find the Fisher information for a function of a parameter, $I(g(\theta))$?

Question

If $I(\theta)$ is the Fisher Information for $\theta$

Then how do I find $J=I(g(\theta ))$ ?

Currently my thoughts are that:

$$I(\theta) = E\left( \left(\frac{\partial}{\partial \theta}\log f(X;\theta)\right)^2 \right)$$ and if $u=g(\theta)$ where $g( \cdot )$ is differentiable, then $$\frac{\partial}{\partial u}=\frac{\partial\theta}{\partial u} \frac{\partial}{\partial \theta} =\frac{1}{g'(\theta)}\frac{\partial}{\partial \theta}$$ So $$I(u) = E\left( \left(\frac{\partial}{\partial u}\log f(X;u)\right)^2 \right) = E\left( \frac{1}{(g'(\theta))^2}\left(\frac{\partial}{\partial \theta}\log f(X;\theta)\right)^2 \right) \ \ \text{as u is a function of} \ \theta$$ but at this point I don't see the justification for taking the $\frac{1}{(g'(\theta))^2}$ out of the bracket...

I'm guessing the result should arrive at

$$ I(g(\theta))=\frac{I(\theta)}{(g'(\theta))^2}$$

but I'm not sure how to get this...

Answer 1

The expectation operator is with respect to $X$ and $g'(\theta)$ is constant with respect to $X$ so it can be taken out of the operator -- giving the required result.

Answer 2

Technically, this formula should be stated using two different Fisher informations on the two sides: on one side we have the Fisher information associated to estimating $\theta$ from samples of $X\sim p_\theta$, and on the other side the Fisher information associated to estimating $f(\theta)$ from the same $X$ instead.

Say we want to estimate $f(\theta)$ from $X\sim p_\theta$, for some pdf $x\mapsto p_\theta(x)$. We can get the associated Fisher by thinking the pdf as a function of $y=f(\theta)$ (assuming $f$ is invertible and differentiable around $\theta)$, and thus compute $$I_y(y) = \mathbb{E}[(\partial_y \log p)^2] = \frac{\mathbb{E}[(\partial_\theta\log p)^2]}{f'(\theta)^2} = \frac{I_\theta(\theta)}{f'(\theta)^2}.$$ Note that the way to interpret $I_y(y)$ with $y=f(\theta)$ is that it tells you about the task of estimating the parameter $\theta$ from a distribution $p_{f(\theta)}$.

For example, suppose we want to estimate $p$ from $X\sim\operatorname{Bern}(\sqrt p)$, meaning $\mathbb{P}(1|p)=\sqrt p$. Then we get the Fisher $$I(p)= \frac{1}{4p} \frac{1}{p(1-p)},$$ where I used the fact that the Bernoulli distribution has Fisher $I(p)=\frac{1}{p(1-p)}$. Notice how this gets much smaller for $p\approx0$, and corresponds to using the (locally unbiased) estimator $$\hat p =-p + \binom{0}{2\sqrt p}.$$