I believe that $A^TA$ is a key matrix structure because of its connection to variance-covariance matrices, which are used in data analysis (e.g. linear regression or principal component analysis). There are interesting properties to $A^TA$, which I think include symmetry and positive semi-definiteness, and real eigenvalues. Just to share my approximate degree of familiarity with the topic.

However, I am missing at least a rung in the ladder when attempting to understand this answer in CV, which revolves around the expression:

$$X'X = V D V', \quad D = \begin{bmatrix} d_1 & & \\ & \ddots & \\ & & d_p \end{bmatrix}, d_i \geq 0$$

In particular I would appreciate being tipped-off as to why the following statement is obvious:

"Now since matrix inversion corresponds to inversion of the eigenvalues, the OLS estimator requires $(X′X)^{-1}=VD^{−1}V′$ (note that $V′=V^{−1}$). Obviously this only works if all eigenvalues are strictly greater than zero, $d_i>0$."

  • $\begingroup$ Where did you stuck? It is explained in the excerpt you quoted. Notice that $d_i$'s are the eigen values and the inverse of a diagonal matrix is a diagonal matrix with the inverse of the diagonal components. $\endgroup$ – user251257 Aug 18 '16 at 23:19
  • $\begingroup$ the eigenvalues of $X'X$ can't be negative. If one eigenvalue is zero, the matrix isn't invertible. So Computing the inverse makes no sense. $\endgroup$ – user251257 Aug 18 '16 at 23:25
  • $\begingroup$ @user251257 The fact that a zero eigenvalue makes the matrix singular is true of all square matrices? $\endgroup$ – Antoni Parellada Aug 18 '16 at 23:27
  • 2
    $\begingroup$ yes. Just compare the definitions of singular and eigenvalues. It is straight forward. $\endgroup$ – user251257 Aug 18 '16 at 23:32
  • 1
    $\begingroup$ @AntoniParellada A good way of remembering that fact is that the determinant is equal to the product of the eigenvalues. $\endgroup$ – Math1000 Aug 19 '16 at 0:10

May be some Linear algebra facts/results will make it clearer. From the equality $X'X = VDV'$, where $V$ is orthogonal matrix, hence $V'=V^{-1}$, you deduce that $X'X$ is similar to $D=diag(\lambda_1,...,\lambda_2)$. As such, they have the same eigenvalues. So, because the OLS estimator is defined by $(X'X)^{-1}X'y$, you need to invert the $(X'X)$ and moreover its inverse have to be positive definite matrix in order to minimize the square errors. This stems from the fact that $(X'X)^{-1}$ is the Hessian matrix of the $S(\beta)=(y-X\beta)'(y-X\beta)$. Thus, for $(X'X)$ to be invertible it has to be full-ranked matrix, namely all eigenvalues have to differ from $0$, and for the positive definite property it is required that each its eigenvalue will be strictly positive, i.e. $\lambda_i >0$, $\forall i$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.