8
$\begingroup$

For given linear model $y = x \beta + \epsilon$, where $\beta$ is a $p$-dimentional column vector, and $\epsilon$ is a measurement error that follows a normal distribution, a FIM is a $p \times p$ positive definite matrix.

How to find elements of the matrix?

$\endgroup$

2 Answers 2

16
$\begingroup$

I'm going to assume that the variance $\sigma^2$ is known since you appear to only be considering the parameter vector $\beta$ as your unknowns. If I observe a single instance $(x, y)$ then the log-likelihood of the data is given by the density $$ \ell(\beta)= -\frac 1 2 \log(2\pi\sigma^2) - \frac{-(y-x^T\beta)^2}{2\sigma^2}. $$ This is just the log of the Gaussian density. The Fisher information matrix is just the expected value of the negative of the Hessian matrix of $\ell(\beta)$. So, taking the gradient gives $$ S(\beta) = \nabla_\beta \frac{-(y-x^T\beta)^2}{2\sigma^2} = \nabla_\beta \left[-\frac{y^2}{2\sigma^2} + \frac{yx^T\beta}{\sigma^2} - \frac{\beta^Txx^T\beta}{2\sigma^2}\right] = \frac{yx}{\sigma^2} - \frac{xx^T\beta}{\sigma^2} = \frac{(y - x^T\beta)x}{\sigma^2}. $$ Taking another derivative, the Hessian is $$ H(\beta) = \frac{\partial}{\partial \beta^T} \frac{(y-x^T\beta)x}{\sigma^2} = \frac{\partial xy}{\partial \beta^T} - \frac{\partial xx^T\beta}{\partial \beta^T} = \frac{-xx^T}{\sigma^2}, $$ so the Fisher information is $$ I(\beta) = -E_\beta H(\beta) = \frac{xx^T}{\sigma^2}. $$ Because gradients and Hessians are additive, if I observe $n$ data items I just add the individual Fisher information matrices, $$ I(\beta) = \frac{\sum_i x_ix_i^T}{\sigma^2}, $$ which, if $X^T = (x_1, x_2, \ldots, x_n)$, can be compactly written as $$ I(\beta) = X^TX / \sigma^2. $$ It is well-known that the variance of the MLE $\hat \beta$ in a linear model is given by $\sigma^2 (X^TX)^{-1}$, and in more general settings the asymptotic variance of the MLE should be equal to the inverse of the Fisher information, so we know we've got the right answer.

$\endgroup$
0
0
$\begingroup$

Let $\gamma$ denote the gaussian distribution of $\epsilon$. The likelihood of the model is $$ \gamma(y - x \beta) $$ where $y$ is your observation and $\beta$ is the parameter. You can now apply the definition of the Fisher Information matrix, $$ I = \text{var} \left( \nabla_\beta \log \gamma(Y - x\beta) \right).$$

$\endgroup$
1
  • $\begingroup$ Thanks for your reply. Still do not understand how the elements on main and sub diagonals will look like? $\endgroup$
    – caspik
    Mar 12, 2013 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.