I've been reading this document. The goal is to find projection of $b$ on the line $L$ which is determined by vector $a$. ($Proj_L(b)$)

Figure 1 in document linked above.

In the document, It is mentioned that If $p$ is thought as approximation to $b$, then $e=b-p$ is the error in that approximation (the word approximation is little confusing though, Isn't $e$ the exact value? Although this is not the problem).

Then we know that If $p$ lies on the line through $a$, then $p = xa$ for some $x$. We also know that, $p$ is orthogonal to $e$, therefore their dot product equates to zero:


$a^Tb - a^Txa = 0$

$xa^Ta = a^Tb$

$x = \frac{a^Tb}{a^Ta}$

Solving for $p$:

$p = ax = a\frac{a^Tb}{xa^Ta}$

First part was almost completely understandable, but in the second part this projection is written in the terms of projection matrix ("$P: p = Pb$"):

$p = xa = \frac{aa^Ta}{a^Ta}$

where did $b$ go? $x$ and $a$ have changed places, but isn't dot product commutative?

Then, $P$ is solved:

$P = \frac{aa^T}{a^Ta}$

Somehow, $aa^T$ is 3x3 matrix.

How was this concluded? From my knowledge, The general definition of projection matrix is $A (A^{T}A)^{-1} A^T \vec{x}$ (where $A$ is matrix)

Does the definition above has any relations with the projection matrix that was represented in document? If not, how was it derived?

Thank you!

  • 1
    $\begingroup$ here $x = \frac{a^Tb}{xa^Ta}$ you have an extra x at the denominator, the issues for be are simply typo or change of symbols $\endgroup$
    – user
    Mar 31 '18 at 19:08
  • 1
    $\begingroup$ To see the connection, try computing $A(A^TA)^{-1}A^T$ when $A$ is $n\times1$. $\endgroup$
    – amd
    Mar 31 '18 at 19:39

The key point is that from here

$$p = ax = a\frac{a^Tb}{a^Ta}$$

we can write in matrix form

$$p = ax = a\frac{a^Tb}{a^Ta}=\frac{aa^T}{a^Ta}b=Pb$$

From here we can generalize for a projection onto a subspace spanned by multiple vectors $a_i$.

Let consider the matrix $A=[a_1 \, a_2\,...\, a_n]$ and the vector $b$ to project then consider


the error is $e=b-p=b-Ax$ and it is miminized when $e$ is orthogonal to $Col(A)$ that is

$$A^Te=A^T(b-Ax)=0\implies A^Tb=A^TAx\implies x=(A^TA)^{-1}A^Tb$$

and then

$$p=Ax=A(A^TA)^{-1}A^Tb=Pb\implies P=A(A^TA)^{-1}A^T$$

  • $\begingroup$ So from my understanding both methods can be used to represent projection as projection matrix? Also apologies for my lack of understanding, but is the second method some specific generalization? $\endgroup$
    – ShellRox
    Mar 31 '18 at 19:59
  • 1
    $\begingroup$ @ShellRox The first case is the particular case of projection on a line (subspace with dimension $=1$, the second case is more general and valid for projection onto subspace ith dimension $\ge 1$. $\endgroup$
    – user
    Mar 31 '18 at 20:04
  • $\begingroup$ Also final question, I've tested these equations (substituting variables with vectors) and I'm unable to understand how is first equation matrix? Maybe these variables should be matrices and not vectors? $\endgroup$
    – ShellRox
    Apr 1 '18 at 11:31
  • 1
    $\begingroup$ $aa^T$ is a matrix and $a^Ta$ is a scalar, try with $a=(1,0,0)$ $\endgroup$
    – user
    Apr 1 '18 at 11:39
  • 1
    $\begingroup$ Pb is the projected vector p, try with some numerical example $\endgroup$
    – user
    Apr 1 '18 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.