8
$\begingroup$

For given linear model $y = x \beta + \epsilon$, where $\beta$ is a $p$-dimentional column vector, and $\epsilon$ is a measurement error that follows a normal distribution, a FIM is a $p \times p$ positive definite matrix.

How to find elements of the matrix?

$\endgroup$

2 Answers 2

16
$\begingroup$

I'm going to assume that the variance $\sigma^2$ is known since you appear to only consider the parameter vector $\beta$ as your unknowns. If I observe a single instance $(x, y)$ then the log-likelihood of the data is given by the density $$ \ell(\beta)= -\frac 1 2 \log(2\pi\sigma^2) - \frac{(y-x^T\beta)^2}{2\sigma^2}. $$ This is just the log of the Gaussian density. The Fisher information matrix is just the expected value of the negative of the Hessian matrix of $\ell(\beta)$. So, taking the gradient gives $$ S(\beta) = \nabla_\beta \frac{-(y-x^T\beta)^2}{2\sigma^2} = \nabla_\beta \left[-\frac{y^2}{2\sigma^2} + \frac{yx^T\beta}{\sigma^2} - \frac{\beta^Txx^T\beta}{2\sigma^2}\right] = \frac{yx}{\sigma^2} - \frac{xx^T\beta}{\sigma^2} = \frac{(y - x^T\beta)x}{\sigma^2}. $$ Taking another derivative, the Hessian is $$ H(\beta) = \frac{\partial}{\partial \beta^T} \frac{(y-x^T\beta)x}{\sigma^2} = \frac{\partial xy}{\partial \beta^T} - \frac{\partial xx^T\beta}{\partial \beta^T} = \frac{-xx^T}{\sigma^2}, $$ so the Fisher information is $$ I(\beta) = -E_\beta H(\beta) = \frac{xx^T}{\sigma^2}. $$ Because gradients and Hessians are additive, if I observe $n$ data items I just add the individual Fisher information matrices, $$ I(\beta) = \frac{\sum_i x_ix_i^T}{\sigma^2}, $$ which, if $X^T = (x_1, x_2, \ldots, x_n)$, can be compactly written as $$ I(\beta) = X^TX / \sigma^2. $$ It is well-known that the variance of the MLE $\hat \beta$ in a linear model is given by $\sigma^2 (X^TX)^{-1}$, and in more general settings the asymptotic variance of the MLE should be equal to the inverse of the Fisher information, so we know we've got the right answer.

$\endgroup$
0
0
$\begingroup$

Let $\gamma$ denote the gaussian distribution of $\epsilon$. The likelihood of the model is $$ \gamma(y - x \beta) $$ where $y$ is your observation and $\beta$ is the parameter. You can now apply the definition of the Fisher Information matrix, $$ I = \text{var} \left( \nabla_\beta \log \gamma(Y - x\beta) \right).$$

$\endgroup$
1
  • $\begingroup$ Thanks for your reply. Still do not understand how the elements on main and sub diagonals will look like? $\endgroup$
    – caspik
    Commented Mar 12, 2013 at 22:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .