4
$\begingroup$

I'm having a bit of trouble tying two related ideas together, and I think I'm just missing a silly detail somewhere.

In the standard formulation of linear least-squares, it can be shown that given a full-rank matrix $A$, one can form the projection matrix $$ P_A = A(A^TA)^{-1}A^T $$

which acts on vectors by orthogonally projecting them onto the column space of $A$.

On the other hand, given a subspace $S \subseteq V$ of a vector space, where $S = \mathrm{Span}(\mathbf u_1, \mathbf u_2, \cdots \mathbf u_k)$ and the $\mathbf u_i$ form an orthonormal basis of $S$, then one can construct a matrix with each $\mathbf u_i$ in the $i$th column: $$ U = [\mathbf u_1, \mathbf u_2, \cdots \mathbf u_k]. $$

One can then form the projection $$ P_S = UU^T $$ which projects vectors in $V$ onto the subspace $S$.

I have two questions:

  1. The matrix $U$ seems to be orthogonal, which would seem to mean it satisfies $UU^T = I$. What exactly am I missing?

  2. The column space of $U$ is $S$ by construction, so how would one recover the formula for $P_S$ by letting $A=U$ in the formula for $P_A$? If $U$ is orthogonal, I see how $(A^TA)^{-1} = I$, but by the first question, it would also seem to reduce the remaining $AA^T$ to $I$ as well.

$\endgroup$
3
$\begingroup$

If $k$ is less than the dimension of $V$, then while $U^T U$ will be an identity matrix (because the columns of $U$ are orthonormal), $UU^T$ will generally not be an identity matrix (because there will be no guarantee that the rows of $U$ are orthonormal. In fact, you can show that if $U$ has size $n\times k$ where $n > k$, then $UU^T$ can never be the identity matrix.)

For example, suppose $$ U = \begin{bmatrix} 1/\sqrt{2} & 0 \\ 1/\sqrt{2} & 0\\ 0 & 1\end{bmatrix}. $$

Then $U^TU = I$, while $$ \begin{align*} UU^T &= \begin{bmatrix} 1/\sqrt{2} & 0 \\ 1/\sqrt{2} & 0\\ 0 & 1\end{bmatrix}\begin{bmatrix} 1/\sqrt{2} & 1/\sqrt{2} & 0\\ 0 & 0 & 1\end{bmatrix} \\ &= \begin{bmatrix}1/2 & 1/2 & 0\\ 1/2 & 1/2 & 0 \\ 0 & 0& 1\end{bmatrix}. \end{align*} $$

$\endgroup$
  • $\begingroup$ Excellent, thank you! That's exactly what I was missing, I was assuming a square matrix and all of my examples were square. $\endgroup$ – D. Zack Garza Feb 18 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.