1
$\begingroup$

I think the statement is true, since it is how regularization works in linear model (where adding $\lambda I$ to $X^TX$ make it full rank). But why?

$\endgroup$
  • 1
    $\begingroup$ Not any real matrix. Obviously it's not true of $-\mathbf{I}$. $\endgroup$ – eyeballfrog May 31 '17 at 2:49
  • 1
    $\begingroup$ @eyeballfrog thanks. how about other cases? $\endgroup$ – hxd1011 May 31 '17 at 2:50
4
$\begingroup$

$X^TX$ is a symmetric matrix. It is diagonalizable with real eigenvalues.

Let those eigenvalues equal $\lambda_1, \cdots \lambda_n$

If we choose any of the $\lambda_i$ above.

$(X^TX - \lambda_i I)$ will create a singular matrix.

However, $(X^TX - \lambda I)$ where $\lambda \ne \lambda_1, \cdots \lambda_n$

then the eigenvalues of $(X^TX - \lambda I)$ equal $\lambda_1 - \lambda,\cdots,\lambda_n-\lambda$ none of which are equal to $0.$ Hence, $(X^TX - \lambda I)$ is non-singular.

Nearly any perturbation to a singular matrix will make for a non-singular matrix.

$\endgroup$
  • $\begingroup$ thanks!!, I have many years of mis-understanding, and your answer helped me a lot $\endgroup$ – hxd1011 May 31 '17 at 2:59
  • $\begingroup$ could you explain more on the last statement? I know why $(X^TX - \lambda I)$ is full rank, but why $(X^TX + \lambda I)$ is also full rank? $\endgroup$ – hxd1011 May 31 '17 at 3:07
1
$\begingroup$

For any square matrix $A$ such that $A$ is not invertible, and for all $n\in\mathbb{N}$, $A+\tfrac{1}{n}I$ is in fact invertible. This technique is used to show that invertible matrices are dense in the set square of matrices.

Of course, it is not true for all matrices, as shown by the example $-I$ and $I$.

Many theorems about the general linear group related to this are really interesting. In particular, it is possible to show that $\mathrm{GL}(n)$ has two connected components! You should see Rudin's book on analysis for this.

This is also of interest for numerical analysis since every singular matrix can be approximated by a "close to singular" matrix.

I hope you find this interesting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.