# Get a Fisher information matrix for linear model with the normal distribution for measurement error?

For given linear model $y = x \beta + \epsilon$, where $\beta$ is a $p$-dimentional column vector, and $\epsilon$ is a measurement error that follows a normal distribution, a FIM is a $p \times p$ positive definite matrix.

How to find elements of the matrix?

-

I'm going to assume that the variance $\sigma^2$ is known, since you appear to only be considering the parameter vector $\beta$ as your unknowns. If I observe a single instance $(x, y)$ then the log-likelihood of the data is given by the density $$\ell(\beta)= -\frac 1 2 \log(2\pi\sigma^2) - \frac{-(y-x^T\beta)^2}{2\sigma^2}.$$ This is just the log of the Gaussian density. The Fisher information matrix is just the expected vaue of the negative of the Hessian matrix of $\ell(\beta)$. So, taking the gradient gives $$S(\beta) = \nabla_\beta \frac{-(y-x^T\beta)^2}{2\sigma^2} = \nabla_\beta \left[-\frac{y^2}{2\sigma^2} + \frac{yx^T\beta}{\sigma^2} - \frac{\beta^Txx^T\beta}{2\sigma^2}\right] = \frac{yx}{\sigma^2} - \frac{xx^T\beta}{\sigma^2} = \frac{(y - x^T\beta)x}{\sigma^2}.$$ Taking another derivative, the Hessian is $$H(\beta) = \frac{\partial}{\partial \beta^T} \frac{(y-x^T\beta)x}{\sigma^2} = \frac{\partial xy}{\partial \beta^T} - \frac{\partial xx^T\beta}{\partial \beta^T} = \frac{-xx^T}{\sigma^2},$$ so the Fisher information is $$I(\beta) = -E_\beta H(\beta) = \frac{xx^T}{\sigma^2}.$$ Because gradients and Hessians are additive, if I observe $n$ data items I just add the individual Fisher information matricies, $$I(\beta) = \frac{\sum_i x_ix_i^T}{\sigma^2},$$ which, if $X^T = (x_1, x_2, \ldots, x_n)$, can be compactly written as $$I(\beta) = X^TX / \sigma^2.$$ It is well-known that the variance of the MLE $\hat \beta$ in a linear model is given by $\sigma^2 (X^TX)^{-1}$, and in more general settings the asymptotic variance of the MLE should be equal to the inverse of the Fisher information, so we know we've got the right answer.

-

Let $\gamma$ denote the gaussian distribution of $\epsilon$. The likelihood of the model is $$\gamma(y - x \beta)$$ where $y$ is your observation and $\beta$ is the parameter. You can now apply the definition of the Fisher Information matrix, $$I = \text{var} \left( \nabla_\beta \log \gamma(Y - x\beta) \right).$$

-
Thanks for your reply. Still do not understand how the elements on main and sub diagonals will look like? – caspik Mar 12 '13 at 22:10