0
$\begingroup$

I am trying to resolve my confusion regarding vector projection and projected vectors with projection matrices.

Suppose the column vectors of $\mathbf{G}$ are orthonormal. To my understanding the vector projection of vector $\mathbf{a}$ onto vector $\mathbf{b}$ is $\hat{\mathbf{a}}(\frac{\mathbf{a}\cdot\mathbf{b}}{||\mathbf{a}||||\mathbf{b}||}$). Then how come in, say, PCA the projection of a data matrix $\mathbf{X}$ onto the new orthonormal basis vectors of $\mathbf{G}$ is given by $\mathbf{G}^T(\mathbf{X} - \mathbb{E}(\mathbf{X}))$? Specifically, the first column vector of $\mathbf{X}$ ends up being $\mathbf{G}^T(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)) = \sum_i\mathbf{g}_i^T(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1))$, where the $i$th component of the projected version of $(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)$ is given by the dot product with the $i$th column vector of $\mathbf{G}$. But at what point are we dividing with the length of the vector $(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)$ like in the vector projection? Or is there a reason why this is not necessary?

$\endgroup$
1
  • 1
    $\begingroup$ The projection onto $b$ should be in the subspace generated by $b$. The projection formula you've provided doesn't meet that criteria. I think you're confusing the formula for $\cos \theta$ with the projection formula. $\endgroup$ Mar 6 '21 at 18:48
1
$\begingroup$

Suppose that $\mathbf G$ has orthonormal columns $\mathbf g_1,\dots,\mathbf g_k$. Because the columns are orthonormal, we can get the projection onto the span of these vectors by adding up individual projections. That is, the projection a vector $\mathbf a$ is given by $$ \operatorname{proj}_{\mathbf G}(\mathbf a) = \frac{\mathbf g_1\cdot \mathbf a}{\|\mathbf g_1\|^2} \mathbf g_1 + \cdots + \frac{\mathbf g_k \cdot \mathbf a}{\|\mathbf g_k\|^2}\mathbf g_k \\ = (\mathbf g_1 \cdot \mathbf a)\mathbf g_1 + \cdots + (\mathbf g_k \cdot \mathbf a)\mathbf g_k. $$ The coefficients of this sum are $\mathbf g_1\cdot \mathbf a, \dots, \mathbf g_k \cdot \mathbf a$, which are the entries of the product $\mathbf G^T\mathbf a.$ To get the corresponding projected vector, take the corresponding linear combination of the columns of $\mathbf G$. In other words, we have $$ \operatorname{proj}_{\mathbf G}(\mathbf a) = \mathbf G(\mathbf G^T \mathbf a). $$ With that in mind, we can find the coefficients of the projection for each column of the matrix $\mathbf M = \mathbf X - \mathbb E(\mathbf X)$ by computing $\mathbf G^T\mathbf M$ and obtain the vectors corresponding to these columns of coefficients with the product $\mathbf G\mathbf G^T\mathbf M$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.