14
$\begingroup$

I believe that $A^TA$ is a key matrix structure because of its connection to variance-covariance matrices.

In Professor Strang's linear algebra lectures, "A-transpose-A" - with this nomenclature, as opposed to $X'X$, for example - is the revolving axis.

Yet, it is not easy to find on a quick Google search a list of its properties. I presume that part of the reason may be that they are shared by variance-covariance matrices. But I'd like to confirm this (does it have identical properties to a var-cov matrix?), and have the list easily available from now on here at SE-Mathematics.

Just to not shy away from the initial effort, here is what I think I have so far:

  1. Symmetry
  2. Positive semidefinite-ness
  3. Real and positive eigenvalues
  4. The trace is positive (the trace is the sum of eigenvalues)
  5. The determinant is positive (the determinant is the product of the eigenvalues)
  6. The diagonal entries are all positive
  7. Orthogonal eigenvectors (**)
  8. Diagonalizable as $Q\Lambda Q^T$
  9. It is possible to obtain a Cholesky decomposition.
  10. Rank of $A^TA$ is the same as rank of $A$.
  11. $\text{ker}(A^TA)=\text{ker}(A)$

(**) The eigenvectors of A-transpose-A form the matrix $V$ in singular value decomposition (SVD) of $A,$ while the square root of the eigenvalues of A-transpose-A are the singular values of the SVD. Similarly, the eigenvectors of A-A-transpose $AA^\top$ include the columns in the matrix $U$ of the SVD of $A.$ The importance of this is exemplified in the fact that SVD can be used to solve least squares regression by computing the Penrose-Moore pseudo-inverse $A^\dagger = V\Sigma^\dagger U^*,$ although the QR decomposition is a more expedient computational method.

There is a nice post on the topic here.

$\endgroup$
5
  • $\begingroup$ Item 1 and 2 are straight forward. Item is a result due to symmetry. It is usually discussed before or in context of eigenvalue decomposition or Jordan normal form $\endgroup$
    – user251257
    Commented Aug 18, 2016 at 23:22
  • $\begingroup$ It is a key matrix structure because of the role it plays in orthogonal projection. Covariance matrices are just special cases. $\endgroup$
    – symplectomorphic
    Commented Aug 18, 2016 at 23:51
  • 1
    $\begingroup$ This is known as a Gramian matrix en.wikipedia.org/wiki/Gramian_matrix $\endgroup$
    – Christian Chapman
    Commented Aug 19, 2016 at 0:44
  • $\begingroup$ Equivalently, you are asking for a list of properties of symmetric positive semidefinite matrices (because every s.p.s.d. matrix can be written as $A^TA$ for some $A$). A bunch of such properties are listed on the Wikipedia page. $\endgroup$
    – user856
    Commented Aug 19, 2016 at 1:33
  • $\begingroup$ @symplectomorphic I have been intrigued by your comment for a while now, and would like to get some reference summarizing this issue. For instance, it is immediate to think about ordinary linear regression and PCA, both being forms of projection (over column space of the model matrix, or on the eigenvectors of the covariance). However, I am shooting for something more general, more conceptual... $\endgroup$
    – Antoni Parellada
    Commented Sep 7, 2016 at 21:25

1 Answer 1

Reset to default
7
$\begingroup$

Yes, it has all the properties of a covariance matrix because it is one. You can define a multivariate normal distribution for which $A^T A$ is the covariance matrix.

$\endgroup$
4
  • 1
    $\begingroup$ Can we spell out the list of all the properties? $\endgroup$
    – Antoni Parellada
    Commented Aug 18, 2016 at 23:29
  • 1
    $\begingroup$ You can't literally list all the properties, because there are infinitely many. Of course, many of those are trivial consequences of other properties, others so obscure that nobody would ever care about them. But everything follows pretty much from the orthogonal diagonalization: $A^T A = U^T D U$ where $U$ is a real orthogonal matrix and $D$ is diagonal with nonnegative diagonal elements. $\endgroup$
    – Robert Israel
    Commented Aug 18, 2016 at 23:35
  • $\begingroup$ @RobertIsrael Could you describe how this construction (of distributions) can be done, or point me to a reference? Thanks. $\endgroup$
    – flow2k
    Commented Sep 30, 2023 at 19:58
  • 1
    $\begingroup$ If $Z$ is a vector whose entries are iid standard normal random variables, $A^T Z$ has mean $0$ and covariance matrix $A^T A$. $\endgroup$
    – Robert Israel
    Commented Oct 2, 2023 at 1:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .