$\bf A^TA$ forms (or equivalently (?) positive semidefinite matrices, or more particularly, covariance matrices($\bf \Sigma$)) are linked in practice to many operations in which data points are orthogonally projected:

  1. In ordinary linear regression (OLS) is part of the projection matrix $\bf P = X(\color{blue}{X^TX})^{−1}X^T$ of the "dependent variable" on the column space of the model matrix.

  2. In principal component analysis (PCA) the data is projected on the eigenvectors of the covariance matrix.

  3. The covariance matrix informs white random "white" samples into diagonal projections in Gaussian processes, which seems intuitively to correspond to a way of projecting.

But I am looking at a unifying explanation. A more generic concept.

In this regard, I have come across the sentence, "It is as if the covariance matrix stored all possible projection variances in all directions," a statement seemingly supported by the fact that a for data cloud in $\mathbb R^n$, the variance of the projection of the points onto a unit vector $\bf u$ will be given by $\bf u^T \Sigma u$.

So is there a way of unify all these inter-related properties into a single set of principles from which all the applications and geometric derivations can be seen?

I believe that the unifying theme is related to the the orthogonal diagonalization $\bf A^T A = U^T D U$ as mentioned here, but I'd like to see this idea explained a bit further.


It was far from self-evident, but after some help by Michael Hardy and @stewbasic, the answer by Étienne Bézout may be starting to click. So like in the move Memento, I'd better tattoo what I got so far here in case it is blurry in the morning:

Concept One:

Block matrix multiplication:

\begin{align} A^\top A & = \begin{bmatrix} \vdots & \vdots & \vdots & \cdots & \vdots \\ a_1^\top & a_2^\top & a_3^\top & \cdots & a_{\color{blue}{\bf n}}^\top\\ \vdots & \vdots & \vdots & \cdots & \vdots\end{bmatrix} \begin{bmatrix} \cdots & a_1 & \cdots\\ \cdots & a_2 & \cdots \\ \cdots & a_3 & \cdots \\ & \vdots&\\ \cdots & a_{\color{blue}{\bf n}} & \cdots \end{bmatrix}\\ &= a_1^\top a_1 + a_2^\top a_2 + a_3^\top a_3 + \cdots+a_n^\top a_n\tag{1} \end{align}

where $a_i$'s are $[\color{blue}{1 \times \bf n}]$ row vectors.

Concept Two:

The $\color{blue}{\bf n}$.

We have the same dimensions for the block matrix multiplication $\bf \underset{[\text{many rows} \times \color{blue}{\bf n}]}{\bf A^\top}\underset{[\color{blue}{\bf n} \times \text{many rows}]}{\bf A} =\large [\color{blue}{\bf n} \times \color{blue}{\bf n}] \small \text{ matrix}$, as for each individual summand $\bf a_i^\top a_i$ in Eq. 1.

Concept Three:

$\bf a_i^\top a_i$ is deceptive because of the key definition: row vector.

Because $\bf a_i$ was defined as a row vector, and the $\bf a_i$ vectors are normalized ($\vert a_i \vert =1$), $\bf a_i^\top a_i$ is really a matrix of the form $\bf XX^\top$, which is a projection matrix provided the $a_i$ vectors are independent (check: "...are linearly independent"), and orthonormal (not a requisite in the answer ("I'm no longer saying they are orthogonal")) - $\color{red}{\text{Do these vectors actually need to be defined as orthonormal?}}$ Or can this constraint of orthonormality of the vectors $a_i$ be relaxed, or implicitly fulfilled by virtue of other considerations? Otherwise we have a rather specific $\bf A$ matrix, making the results less generalizable.

Concept Four:

A projection onto what?

Onto the subspace spanned by the column space of $\bf X$ (think OLS projection ${\bf A}\color{gray}{(A^\top A)^{-1}} {\bf A^\top}$). But what is $\bf X$ here? None other than $\bf a_i^\top$, and since $\bf a_i$ is a row vector, $\bf a_i^\top$ is a column vector.

So we are doing ortho-projections onto the column space of $\bf A^\top$, which is in $\mathbb R^{\color{blue}{\bf n}}$.

I was hoping that the last sentence could have been, "... onto the column space of $\bf A$...

What are the implications?

  • 1
    $\begingroup$ $\bf A^T A$, being a matrix of all dot products of the columns of $\bf A$ encapsulates the "geometry" of $A$. If you have done some differential geometry, this is equivalent to the metric tensor $g_{ij}$ (mathworld.wolfram.com/MetricTensor.html). It is thus not strange that $\bf (A^T A)^{-1}$ plays an important role as $g^{ij}$ does. $\endgroup$ – Jean Marie Sep 7 '16 at 22:58
  • $\begingroup$ Do you agree that $B:=\bf A^T A$ is made of dot products of columns of $A$ ? If these columns make an orthonormal basis $B=I$, idnetity matrix. The farthest $B$ is from $I$ the more work there will be to orthonormalize the whole... $\endgroup$ – Jean Marie Sep 7 '16 at 23:06

Suppose we are given a matrix $\mathrm A$ that has full column rank. Its SVD is of the form

$$\mathrm A = \mathrm U \Sigma \mathrm V^T = \begin{bmatrix} \mathrm U_1 & \mathrm U_2\end{bmatrix} \begin{bmatrix} \hat\Sigma\\ \mathrm O\end{bmatrix} \mathrm V^T$$

where the zero matrix may be empty. Note that

$$\mathrm A \mathrm A^T = \mathrm U \Sigma \mathrm V^T \mathrm V \Sigma^T \mathrm U^T = \mathrm U \begin{bmatrix} \hat\Sigma^2 & \mathrm O\\ \mathrm O & \mathrm O\end{bmatrix} \mathrm U^T$$

can only be a projection matrix if $\hat\Sigma = \mathrm I$. However,

$$\begin{array}{rl} \mathrm A (\mathrm A^T \mathrm A)^{-1} \mathrm A^T &= \mathrm U \Sigma \mathrm V^T (\mathrm V \Sigma^T \mathrm U^T \mathrm U \Sigma \mathrm V^T)^{-1} \mathrm V \Sigma^T \mathrm U^T\\ &= \mathrm U \Sigma \mathrm V^T (\mathrm V \Sigma^T \mathrm \Sigma \mathrm V^T)^{-1} \mathrm V \Sigma^T \mathrm U^T\\ &= \mathrm U \Sigma \mathrm V^T (\mathrm V \hat\Sigma^2 \mathrm V^T)^{-1} \mathrm V \Sigma^T \mathrm U^T\\ &= \mathrm U \Sigma \mathrm V^T \mathrm V \hat\Sigma^{-2} \mathrm V^T \mathrm V \Sigma^T \mathrm U^T\\ &= \mathrm U \Sigma \hat\Sigma^{-2} \Sigma^T \mathrm U^T\\ &= \mathrm U \begin{bmatrix} \mathrm I & \mathrm O\\ \mathrm O & \mathrm O\end{bmatrix} \mathrm U^T = \mathrm U_1 \mathrm U_1^T\end{array}$$

is always a projection matrix.

| cite | improve this answer | |
  • $\begingroup$ Would you mind giving some context to the expression $\begin{bmatrix} \hat\Sigma\\ \mathrm O\end{bmatrix}$? $\endgroup$ – Antoni Parellada Sep 8 '16 at 0:35
  • $\begingroup$ @AntoniParellada If $\mathrm A$ has full column rank, then it's either square or thin (it cannot be fat) and all its singular values are positive. Matrix $\hat\Sigma$ is square, diagonal and positive. $\endgroup$ – Rodrigo de Azevedo Sep 8 '16 at 0:37
  • $\begingroup$ What the notation signifies is then, for example, in the matrix $\tiny\begin{bmatrix} 1&6\\2&7\\3&8\\4&9\\5&10 \end{bmatrix}$ (full rank), upon diagonalizing you end up with the diagonal $\tiny\begin{bmatrix} 19.5&0\\0&1.8\\0&0\\0&0\\0&0 \end{bmatrix}$... The zeros at the bottom are the $\bf O$? $\endgroup$ – Antoni Parellada Sep 8 '16 at 0:44
  • $\begingroup$ @AntoniParellada I wouldn't call it diagonalizing, as the matrix is thin, not square. Note that the columns of $\mathrm U$ are the left singular vectors of $\mathrm A$ and that the columns of $\mathrm U_1$ are the left singular vectors of $\mathrm A$ that span the column space of $\mathrm A$. $\endgroup$ – Rodrigo de Azevedo Sep 8 '16 at 0:51
  • $\begingroup$ Scratch 'diagonalizing' - I meant doing the svd decomposition to get the diagonal $\Sigma$ matrix. I was inquiring about notation - I think it's clear though that the $\mathrm O$ in $\begin{bmatrix} \hat\Sigma\\ \mathrm O\end{bmatrix}$ is meant to represent the rectangular form of $\Sigma$. $\endgroup$ – Antoni Parellada Sep 8 '16 at 13:10

Using block matrix notation, we can write $$A = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \\ \end{bmatrix} $$ and $$A^T = \left[a_1^T a_2^T \dots a_n^T \right], $$ where $a_1,...,a_n$ are the rows of $A$.

Then $A^TA = a_1^Ta_1+\dots a_n^Ta_n$ which is a sum of orthogonal projections on the directions $a_1^T,...,a_n^T,$ if we also assume that $|a_1| = ... = |a_n| = 1$. If $A$ is invertible, then $a_1,...,a_n$ are linearly independent, so $A^TA$ can be seen as a sum of $n$ orthogonal projections on $n$ linearly independent directions in $\mathbb{R}^n.$

This should probably be a comment, but obviously I couldn't fit the equations in that format.

| cite | improve this answer | |
  • $\begingroup$ I see everything you are explaining up to "Of course...". $\endgroup$ – Antoni Parellada Sep 7 '16 at 23:10
  • $\begingroup$ @AntoniParellada Sorry, I was too quick reading your question. My comment/answer assumes that A is orthogonal. I'll change it. $\endgroup$ – Étienne Bézout Sep 7 '16 at 23:17
  • $\begingroup$ The fact that are linearly independent, though, does not imply that they are orthogonal... I feel like there is a missing part in the story, which is probably obvious to you... $\endgroup$ – Antoni Parellada Sep 7 '16 at 23:25
  • 1
    $\begingroup$ The vectors being orthogonal or not does not have any importance for my answer. If $A$ is symmetric, then $a_i^T$ is equal to the $i$ th column of $A$, say $b_i$. Then you get an expression like $A^TA = b_1b_1^T+...+b_nb_n^T$. $\endgroup$ – Étienne Bézout Sep 9 '16 at 17:00
  • 1
    $\begingroup$ @AntoniParellada Ok, no problem. I might think about it a bit more and see if I come up with something. $\endgroup$ – Étienne Bézout Sep 9 '16 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.