2
$\begingroup$

Suppose I have a multiple linear regression model $Y_i = \beta_0 + \beta_1 * X_{i1} + ... \beta_p * X_{ip} + \epsilon_i$ where $\epsilon_i \sim N(0, \sigma^2)$ and $Cov(\epsilon_i, \epsilon_j) = 0$ for $i \neq j$. If two of the predictors are correlated, then that will inflate the variance of the coefficient estimates, leading to invalid inferences. In the MLR setting, the variance of the least squares estimator $\hat{\beta}$ is $\sigma^2(X'X)^{-1}$. Can someone give me some mathematical intuition as to why $\sigma^2(X'X)^{-1}$ will inflate/become unstable if there's multicollinearity?

$\endgroup$
1
$\begingroup$

It's easiest to see this in terms of the eigenvalues of $X'X$. Since $x'X$ is symmetric and real, its eigenvalues are real, and we can diagonalize it as

$X'X=U\Lambda U'$

where $\Lambda$ is a diagonal matrix of the eigenvalues of $X'X$, and $U$ is an orthogonal matrix. Then

$(X'X)^{-1}=U \Lambda^{-1}U'$

If there is multicolinearity, then $X'X$ is nearly singular, or in terms of the eigenvalues, one or more of the eigenvalues is very small or zero. Very small eigenvalues cause $\Lambda^{-1}$ to have very large entries. This in turn causes $(X'X)^{-1}$ to have very large entries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.