2
$\begingroup$

I'm currently studying principal component analysis (PCA) from this lecture notes. I understand that we are trying to find the axis on which the variance of projection of all the data points is maximum.

Now where I'm stuck is in the formulation of PCA. In the above notes, on page number 5, our objective function is given as following.

$$\frac{1}{m}\sum_{i=1}^{m}(x^{(i)^{T}}u)^2 = \frac{1}{m}\sum_{i=1}^{m}u^Tx^{(i)}x^{(i)^T}u$$

How this is derived? I can't find any explanation on this step anywhere. As per my understanding shouldn't it be:

$$\frac{1}{m}\sum_{i=1}^{m}(x^{(i)^{T}}u)^2 = \frac{1}{m}\sum_{i=1}^{m}x^{(i)^T}ux^{(i)^T}u$$

Also in this question, the objective function is very different then the one mentioned above in the notes. (i.e from here and here it seems to be the distance between data point $x^{(i)}$ and the axis $w$, but in the notes and video lectures, it's mentioned that it's distance between projection of point and the origin).

What am I missing here? Any help would be appreciated

$\endgroup$
2
$\begingroup$

Note that ${x^{(i)}}^Tu$ is a scalar, since $x^{(i)}$ and $u$ are 2 vectors in $\Bbb R^n$. If I denote this number $\lambda$, then it corresponds to a $1\times 1$ matrix $[\lambda]$ and for this matrix, we have: $$[\lambda]^T [\lambda] = [\lambda]^2 = [\lambda^2]$$

To conclude, note that $[\lambda]^T[\lambda] = ({x^{(i)}}^Tu)^T ({x^{(i)}}^Tu) = u^T x^{(i)}{x^{(i)}}^Tu$.

$\endgroup$
4
  • $\begingroup$ $x^{(i)^T}u$ is a dot product right? $\endgroup$
    – Kaushal28
    Jul 10 '18 at 17:55
  • $\begingroup$ You're perfectly right. $\endgroup$
    – paf
    Jul 10 '18 at 18:00
  • 1
    $\begingroup$ Okay that's right. But how about my last paragraph of the question? Which is the correct representation of this formula? It's distance between origin and projection or distance between data point and the line? +1 for the explanation btw. $\endgroup$
    – Kaushal28
    Jul 10 '18 at 18:21
  • $\begingroup$ Sorry for the delay. $x^{(i)^T}u$ is the length of the projection of $x^{(i)}$ on the line generated by $u$: mathworld.wolfram.com/DotProduct.html $\endgroup$
    – paf
    Jul 12 '18 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.