Suppose we have a linear subspace $S = \operatorname{span}(w_1, \ldots, w_m) \subset \mathbb R^{n}$, where $w_1, \ldots, w_m$ form an orthonormal basis for $S$. Given $x \in \mathbb R^n$, denote by $W = (w_1, \ldots, w_m)$ the matrix whose column vectors are $w_i$, the projection of $x$ onto $S$ is given by $W^{T}W x$. I wonder how one proves this through the formula: $Px = \sum_{i} \langle w_i, x \rangle w_i$.


1 Answer 1


So your premises are wrong here. The core thing is to remember orthonormality. W^T * W = I, as when i = j, <w_i, w_j> = 1, and when i != j, <w_i, w_j> = 1. This would imply that the projection of x here is just x, which isn't accurate in all cases, so your provided equation is wrong

What you are looking for instead is the least squares matrix solution, which is equivalent to a project onto a set of columns: x_hat = (W^T * W)^(-1)* W^T * x satisfies the least squares problem min_x_hat ||W * x_hat - x||^2, which would be a projection onto W

Note that the inverse term is just the identity due to that orthonormality properly. x_hat here is the best estimate you can get for x in the basis of W.

This gives us an estimate of x_hat = [<w_1, x>, ..., <w_m, x>]^T

To view x_hat in the standard basis, you take W * x_hat = Px as you defined above

You can also view this in terms of Gram Schmidt, a process that generates orthogonal vectors, which shows that for any subspace you can find an orthonormal basis, where the computation looks very similar to Px.

  • $\begingroup$ Sorry I meant to say $W^T W$ $\endgroup$
    – koch
    Jul 15, 2020 at 0:17
  • $\begingroup$ That is still wrong. It should be W * W^T * x =. W * [<w_1, x>, ..., <w_m, x>]^T = [w_1 ... w+m] * [<w_1, x>, ..., <w_m, x>]^T = <w_1, x> * w_1 + ... + <w_m, x> * w_m = Px. Remember W^T * W is not W * W^T $\endgroup$
    – cmitch
    Jul 15, 2020 at 0:21
  • $\begingroup$ Thank you!! One thing that I am not quite understanding is that you are referring to $\hat x$ as the project and $W \hat x$ as it in standard basis. What is the distinction between the two? I think $W^T \hat x$ is just the coefficient of the projection. But many resources does consider it as the actual projection. $\endgroup$
    – koch
    Jul 15, 2020 at 2:53
  • $\begingroup$ So it is all about what basis you are working in. To have $x$ in the standard basis means $x = x_1 e_1 + ... + x_n e_n$ with $e_i$ being the $i^{th}$ standard basis vector. That x_hat $\hat{x}$ is a notation used in optimization such like with least squares, but the core idea is that $x$ in the $W$ basis $xw$ is such that $x = xw_1 w_1 + ... + xw_n w_n$. So it is a projection, but that vector is written relative to $W$. From these equations, we can factor out the coefficients and get $W xw = x$. For other bases other than standard (e.g. a basis U), $W xw = U xu$ $\endgroup$
    – cmitch
    Jul 15, 2020 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.