When doing multiple linear regression,

$\boldsymbol{Y} = X\boldsymbol{\beta}+\boldsymbol{\epsilon}$


$\boldsymbol{\epsilon} \sim N(0, \Sigma)$

The best estimates of the coefficients can be found using the generalised least squares formula:

$\boldsymbol{\hat{\beta}} = (X'\Sigma^{-1}X)^{-1}X'\Sigma^{-1}\boldsymbol{Y}$.

From this we can find:

$Var(\boldsymbol{\hat{\beta}}) = (X'\Sigma^{-1}X)^{-1}$

Alternatively, we know that the asymptotic distribution of the estimators is given by

$\boldsymbol{\hat{\beta}} \sim N(\boldsymbol{\beta}, I^{-1})$

where I is the information matrix, and hence

$Var(\boldsymbol{\hat{\beta}}) = I^{-1}$

I have been working through a problem and have been calculating the best estimates of the coefficients using two methods, either by maximising the log-likelihood numerically, or by using generalised least squares (GLS). In both cases I get the same estimates for the parameters, however the covariance matrices are very different, ie I would expect

$(X'\Sigma^{-1}X)^{-1} \approx I^{-1}$

however I am finding that the variances are much smaller when using the Information matrix. This is obviously causing issues when doing hypothesis tests etc. I am simulating my own data and so know what to expect when doing these tests, and it works nicely when using GLS, but the lower variance from the Information matrix is giving me inconsistent results.

Any ideas why this is happening?

  • $\begingroup$ @ A.E which part do you think is not right? $\endgroup$ – darren86 Oct 4 '16 at 13:10
  • $\begingroup$ @ A.E, Apologies for the confusion, but as stated in the original post, I denotes the fisher information matrix, NOT a identity matrix. $\endgroup$ – darren86 Oct 4 '16 at 13:47
  • $\begingroup$ Ok. My bad. Some questions for further clarification: 1) The noise terms are iid normal r.v? 2) When you say that $(X'\Sigma^{-1}X)^{-1}$ is greater then $I^{-1}$ do you mean component wise? By the diagonal entries only? $\endgroup$ – V. Vancak Oct 4 '16 at 14:00
  • $\begingroup$ @A.E 1. The error terms are correlated here, hence why generalised least squares is being used as opposed to ordinary least squares. 2. Although I have not checked all components as its a large matrix, they are generally larger component wise. $\endgroup$ – darren86 Oct 4 '16 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.