# Eigenvalues of the inverse of $A^TA$ strictly positive.

I believe that $A^TA$ is a key matrix structure because of its connection to variance-covariance matrices, which are used in data analysis (e.g. linear regression or principal component analysis). There are interesting properties to $A^TA$, which I think include symmetry and positive semi-definiteness, and real eigenvalues. Just to share my approximate degree of familiarity with the topic.

However, I am missing at least a rung in the ladder when attempting to understand this answer in CV, which revolves around the expression:

$$X'X = V D V', \quad D = \begin{bmatrix} d_1 & & \\ & \ddots & \\ & & d_p \end{bmatrix}, d_i \geq 0$$

In particular I would appreciate being tipped-off as to why the following statement is obvious:

"Now since matrix inversion corresponds to inversion of the eigenvalues, the OLS estimator requires $(X′X)^{-1}=VD^{−1}V′$ (note that $V′=V^{−1}$). Obviously this only works if all eigenvalues are strictly greater than zero, $d_i>0$."

• Where did you stuck? It is explained in the excerpt you quoted. Notice that $d_i$'s are the eigen values and the inverse of a diagonal matrix is a diagonal matrix with the inverse of the diagonal components. – user251257 Aug 18 '16 at 23:19
• the eigenvalues of $X'X$ can't be negative. If one eigenvalue is zero, the matrix isn't invertible. So Computing the inverse makes no sense. – user251257 Aug 18 '16 at 23:25
• @user251257 The fact that a zero eigenvalue makes the matrix singular is true of all square matrices? – Antoni Parellada Aug 18 '16 at 23:27
• yes. Just compare the definitions of singular and eigenvalues. It is straight forward. – user251257 Aug 18 '16 at 23:32
• @AntoniParellada A good way of remembering that fact is that the determinant is equal to the product of the eigenvalues. – Math1000 Aug 19 '16 at 0:10

May be some Linear algebra facts/results will make it clearer. From the equality $X'X = VDV'$, where $V$ is orthogonal matrix, hence $V'=V^{-1}$, you deduce that $X'X$ is similar to $D=diag(\lambda_1,...,\lambda_2)$. As such, they have the same eigenvalues. So, because the OLS estimator is defined by $(X'X)^{-1}X'y$, you need to invert the $(X'X)$ and moreover its inverse have to be positive definite matrix in order to minimize the square errors. This stems from the fact that $(X'X)^{-1}$ is the Hessian matrix of the $S(\beta)=(y-X\beta)'(y-X\beta)$. Thus, for $(X'X)$ to be invertible it has to be full-ranked matrix, namely all eigenvalues have to differ from $0$, and for the positive definite property it is required that each its eigenvalue will be strictly positive, i.e. $\lambda_i >0$, $\forall i$.