39
$\begingroup$

I am trying to understand how - exactly - I go about projecting a vector onto a subspace.

Now, I know enough about linear algebra to know about projections, dot products, spans, etc etc, so I am not sure if I am reading too much into this, or if this is something that I have missed.

For a class I am taking, the proff is saying that we take a vector, and 'simply project it onto a subspace', (where that subspace is formed from a set of orthogonal basis vectors).

Now, I know that a subspace is really, at the end of the day, just a set of vectors. (That satisfy properties here). I get that part - that its this set of vectors. So, how do I "project a vector on this subspace"?

Am I projecting my one vector, (lets call it a[n]) onto ALL the vectors in this subspace? (What if there is an infinite number of them?)

For further context, the proff was saying that lets say we found a set of basis vectors for a signal, (lets call them b[n] and c[n]) then we would project a[n] onto its signal subspace. We project a[n] onto the signal-subspace formed by b[n] and c[n]. Well, how is this done exactly?..

Thanks in advance, let me know if I can clarify anything!

P.S. I appreciate your help, and I would really like for the clarification to this problem to be somewhat 'concrete' - for example, something that I can show for myself over MATLAB. Analogues using 2-D or 3-D space so that I can visualize what is going on would be very much appreciated as well.

Thanks again.

$\endgroup$
  • 1
    $\begingroup$ Hopefully, this can help clear some of your confusion: A projection of a space $V$ onto a subspace $W$ is a map, $\pi:V \to W$, such that $\pi^2=\pi$. So given any vector $v\in V$, we have $\pi(v)\in W$ and $\pi(w)=w$ if $w\in W$. So a projection is a way of associating a vector in a subspace with each vector in the whole space in such a way that vectors in the subspace are associated with themselves. $\endgroup$ – Bill Cook Feb 24 '12 at 3:04
  • 2
    $\begingroup$ @BillCook Thank you for your comment, unfortunately I do not understand how this helps me project onto a subspace. In one hand I have a vector a[n]. In the other hand I have two basis vectors of the subspace, b[n], and c[n]. How do I project a[n] on the subspace formed by those other two? $\endgroup$ – Spacey Feb 24 '12 at 3:11
44
$\begingroup$

I will talk about orthogonal projection here.

When one projects a vector, say $v$, onto a subspace, you find the vector in the subspace which is "closest" to $v$. The simplest case is of course if $v$ is already in the subspace, then the projection of $v$ onto the subspace is $v$ itself.

Now, the simplest kind of subspace is a one dimensional subspace, say the subspace is $U = \operatorname{span}(u)$. Given an arbitrary vector $v$ not in $U$, we can project it onto $U$ by $$v_{\| U} = \frac{\langle v , u \rangle}{\langle u , u \rangle} u$$ which will be a vector in $U$. There will be more vectors than $v$ that have the same projection onto $U$.

Now, let's assume $U = \operatorname{span}(u_1, u_2, \dots, u_k)$ and, since you said so in your question, assume that the $u_i$ are orthogonal. For a vector $v$, you can project $v$ onto $U$ by $$v_{\| U} = \sum_{i =1}^k \frac{\langle v, u_i\rangle}{\langle u_i, u_i \rangle} u_i = \frac{\langle v , u_1 \rangle}{\langle u_1 , u_1 \rangle} u_1 + \dots + \frac{\langle v , u_k \rangle}{\langle u_k , u_k \rangle} u_k.$$

$\endgroup$
  • $\begingroup$ Ah! Yes thanks this is very nice. So simply the new projected vector $p[n] = a[n]^{T}b[n] + a[n]^{T}c[n]$ will be the component of a[n] on the hyper-plane that is the subspace that b[n] and c[n] make together. Is this correct? $\endgroup$ – Spacey Feb 24 '12 at 3:23
  • $\begingroup$ I don't know if your formula for $p[n]$ is correct, but you geometric interpretation of is correct. $\endgroup$ – Calle Feb 24 '12 at 3:26
  • $\begingroup$ The way I currently understand it, I take dot product of a[n] with b[n] and c[n], and I get two scores. Then I take the respective scores and go back and use them as weights on b[n] and c[n], and I add those now weighted basis vectors together. That is the projection of a[n] onto the hyperplane... $\endgroup$ – Spacey Feb 24 '12 at 3:30
  • $\begingroup$ Yes, that is correct. $\endgroup$ – Calle Feb 24 '12 at 3:31
  • 1
    $\begingroup$ @Calle What if $\left\{u_1,\ldots,u_k\right\}$ is not orthogonal but still spans the same subspace? would the projection remain the same, or would I have to go through the Gram-Schmidt first and then project? $\endgroup$ – Jon Jan 7 '17 at 10:18
7
$\begingroup$

Take a basis $\{v_1, \dots, v_n\}$ for the "signal subspace" $V$. Let's assume $V$ is finite dimensional for simplicity and practical purposes, but you can generalize to infinite dimensions. Let's also assume the basis is orthonormal.

The projection of your signal $f$ onto the subspace $V$ is just

$$\mathrm{proj}_V(f) = \sum_{i=1}^n \langle f, v_i \rangle v_i$$

and $f = \mathrm{proj}_V(f) + R(f)$, where $R(f)$ is the remainder, or orthogonal complement, which will be 0 if $f$ lies in the subspace $V$.

The $i$-th term of the sum, $\langle f, v_i\rangle$, is the projection of $f$ onto the subspace spanned by the $i$-th basis vector. (Note, if the $v_i$ are orthogonal, but not necessarily orthonormal, you must divide the $i$-th term by $\|v_i\|^2$.)

$\endgroup$
  • $\begingroup$ Thank you kind sir, that was informative. You said: "The i-th term of the sum, ⟨f,vi⟩, is the projection of f onto the subspace spanned by the i-th basis vector." - and the subspace here is just the set made up of all scalar multiples of the $i-th$ basis, yes? $\endgroup$ – Spacey Feb 24 '12 at 3:39
5
$\begingroup$

The projection of the vector is the vector belonging to the subspace that best approximates the former, i.e. such that the (squared) norm of the difference is the smallest.

WLOG take a subspace spanned by three vectors. We minimize for the coordinates $u,v,w$

$$\epsilon=(\vec d-u\vec a-v\vec b-w\vec c)^2.$$

This is achieved by canceling the gradient (factor $2$ omitted),

$$\begin{cases} \dfrac{d\epsilon}{du}=\vec a\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dv}=\vec b\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dw}=\vec c\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0.\\ \end{cases}$$ giving a linear system of three equations in three unknonws.

In the special case that $\vec a,\vec b,\vec c$ form an orthonormal basis, we simply have

$$u=\vec d\cdot\vec a,v=\vec d\cdot\vec b,w=\vec d\cdot\vec c,$$ and the projection is

$$\vec u^*=(\vec d\cdot\vec a)\vec a+(\vec d\cdot\vec b)\vec b+(\vec d\cdot\vec c)\vec c.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.