1
$\begingroup$

Suppose we have that $X_1, \ldots, X_n$ are iid from a distribution with ONE parameter, $\theta$. Then, under regulatory conditions, the Fisher Information may be written as:

$$ \mathcal{I}(\theta) = - \operatorname{E} \left[\left. \frac{\partial^2}{\partial\theta^2} \log f(X;\theta)\right|\theta \right] $$

I know that for the iid case, we have that:

$$ \mathcal{I}(\theta) = n\mathcal{I}_1(\theta) $$

Where $\mathcal{I}_1(\theta)$ is the Fisher Information for the first sample.

Now, I am wondering why this doesn't hold in a multi-variate setting, where instead of having ONE parameter, we have multiple.

Suppose now that $X_1, \ldots, X_n$ are iid from a distribution with parameters $\theta = (\theta_1, \ldots, \theta_k)$. In this case, it is my understanding that additivity will still hold, that is:

$$ \mathcal{I}(\theta) = \mathcal{I}_1(\theta)+ \ldots + \mathcal{I}_n(\theta) $$

However, why do we not have that case that $\mathcal{I}(\theta) = n\mathcal{I}_1(\theta)$ anymore?

Thanks!

$\endgroup$
1
$\begingroup$

In the multi-parameter case, $\theta = (\theta_1, ..., \theta_k)$, Fisher information is a $k\times k$ matrix where the $ij$ entry is given by $$ \mathcal{I}(\theta)_{ij} = -E[\frac{\partial}{\partial\theta_i \partial \theta_j} \log f(X;\theta)], $$ so, assuming that the sample is i.i.d, the additivity is still holds component-wise in the information matrix.

$\endgroup$
  • $\begingroup$ Thanks, my question relates more to why in some cases we have $n$ times the first fisher information while in other cases we just add? $\endgroup$ – user136503 May 7 '16 at 21:53
  • $\begingroup$ Then it should be an issue of dependence. If your sample is i.i.d, then $I_i(\theta) = I_j(\theta)$, for all $i, j$. Otherwise, if dependence is introduced, then $\sum_{i=1}^nI_i(\theta) \le nI(\theta)$. $\endgroup$ – V. Vancak May 7 '16 at 23:35
  • $\begingroup$ I see, would I also have a problem if I don't have the identically distributed part? $\endgroup$ – user136503 May 7 '16 at 23:52
  • $\begingroup$ sure. as information may depend on the parameters as well, so different parameters will result in different information. $\endgroup$ – V. Vancak May 8 '16 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.