Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For given linear model $y = x \beta + \epsilon$, where $\beta$ is a $p$-dimentional column vector, and $\epsilon$ is a measurement error that follows a normal distribution, a FIM is a $p \times p$ positive definite matrix.

How to find elements of the matrix?

share|cite|improve this question

Let $\gamma$ denote the gaussian distribution of $\epsilon$. The likelihood of the model is $$ \gamma(y - x \beta) $$ where $y$ is your observation and $\beta$ is the parameter. You can now apply the definition of the Fisher Information matrix, $$ I = \text{var} \left( \nabla_\beta \log \gamma(Y - x\beta) \right).$$

share|cite|improve this answer
Thanks for your reply. Still do not understand how the elements on main and sub diagonals will look like? – caspik Mar 12 '13 at 22:10

I'm going to assume that the variance $\sigma^2$ is known, since you appear to only be considering the parameter vector $\beta$ as your unknowns. If I observe a single instance $(x, y)$ then the log-likelihood of the data is given by the density $$ \ell(\beta)= -\frac 1 2 \log(2\pi\sigma^2) - \frac{-(y-x^T\beta)^2}{2\sigma^2}. $$ This is just the log of the Gaussian density. The Fisher information matrix is just the expected vaue of the negative of the Hessian matrix of $\ell(\beta)$. So, taking the gradient gives $$ S(\beta) = \nabla_\beta \frac{-(y-x^T\beta)^2}{2\sigma^2} = \nabla_\beta \left[-\frac{y^2}{2\sigma^2} + \frac{yx^T\beta}{\sigma^2} - \frac{\beta^Txx^T\beta}{2\sigma^2}\right] = \frac{yx}{\sigma^2} - \frac{xx^T\beta}{\sigma^2} = \frac{(y - x^T\beta)x}{\sigma^2}. $$ Taking another derivative, the Hessian is $$ H(\beta) = \frac{\partial}{\partial \beta^T} \frac{(y-x^T\beta)x}{\sigma^2} = \frac{\partial xy}{\partial \beta^T} - \frac{\partial xx^T\beta}{\partial \beta^T} = \frac{-xx^T}{\sigma^2}, $$ so the Fisher information is $$ I(\beta) = -E_\beta H(\beta) = \frac{xx^T}{\sigma^2}. $$ Because gradients and Hessians are additive, if I observe $n$ data items I just add the individual Fisher information matricies, $$ I(\beta) = \frac{\sum_i x_ix_i^T}{\sigma^2}, $$ which, if $X^T = (x_1, x_2, \ldots, x_n)$, can be compactly written as $$ I(\beta) = X^TX / \sigma^2. $$ It is well-known that the variance of the MLE $\hat \beta$ in a linear model is given by $\sigma^2 (X^TX)^{-1}$, and in more general settings the asymptotic variance of the MLE should be equal to the inverse of the Fisher information, so we know we've got the right answer.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.