understanding projection matrices

Question

Let us look at two linearly independent vectors $v_1$ and $v_2$, which span a plane in $\mathbb{R}^n$ through the origin. Let us form a matrix $V$ out of these vectors so that the first column of $V$ is the vector $v_1$ and the second column is the vector $v_2$. Show that the the matrix $P=V(V^TV)^{-1}V^T$ projects all vectors $x\in\mathbb{R}^n$ orthogonally onto the plane spanned by $v_1$ and $v_2$.

Answer 1

Hint: You need to show two things.

1. $P\mathbf{x}$ is in the plane, i.e. $P\mathbf{x}\in\text{span}(\mathbf{v}_1,\mathbf{v}_2)$, for all $\mathbf{x}\in\mathbb{R}^n$
2. $P\mathbf{x}$ (the projection) and $\mathbf{x}-P\mathbf{x}$ are perpendicular, i.e. $\langle \mathbf{x},\mathbf{x}-P\mathbf{x}\rangle=0$.

To help you get started on (1), the point is that we need to find scalars $c_1$ and $c_2$ so that $\mathbf{x}=c_1\mathbf{v}_1+c_2\mathbf{v}_2$. (That is what it means for $\mathbf{x}$ to lie in the plane spanned by $\mathbf{v}_1$ and $\mathbf{v}_2$.) But $$c_1\mathbf{v}_1+c_2\mathbf{v}_2=V\begin{pmatrix}c_1\\c_2\end{pmatrix}=V\mathbf{c}$$ where $\mathbf{c}$ is the vector $(c_1,c_2)\in\mathbb{R}^2$.

So we want to solve the equation $$V\mathbf{c}=\mathbf{x}$$ for $\mathbf{c}$. Now you have to be a little careful: the matrix $V$ is not square...

Answer 2

To show $(1)$, note that $V=[v_1\ v_2]$. Then $$ [Pv_1\ Pv_2]=PV=V(V^TV)^{-1}V^TV=V=[v_1\ v_2]. $$ So $Pv_1=v_1$, $Pv_2=v_2$.

Answer 3

Let $\{ v_1, v_2, w_1, \dots, w_{n-2} \}$ be a basis for V. By Graham-Schmidt, we can choose the $w$'s to be orthogonal to the $v$'s. We can write any x in V as:

$\textbf{x} = (a_1v_1 + a_2v_2) + (b_1w_1 + \dots + b_{n-2}w_{n-2})$.

Or in matrix notation:

$\textbf{x} = V \left( \begin{smallmatrix} a_1 \\ a_2 \end{smallmatrix} \right) + W \left( \begin{smallmatrix} w_1 \\ \vdots \\ w_{n-2} \end{smallmatrix}\right)$

where W is $[w_1 \dots w_{n-2}]$.

Hence $P\textbf{x} = (PV) \left( \begin{smallmatrix} a_1 \\ a_2 \end{smallmatrix} \right) + (PW) \left( \begin{smallmatrix} w_1 \\ \vdots \\ w_{n-2} \end{smallmatrix}\right)$

by linearity and associativity of matrix multiplication. But $PV = V$, and $V^TW$ is zero because it's formed by taking the dot products of v's with w's, which are orthogonal. This makes PW a zero matrix, as well. So

$P\textbf{x} = V \left( \begin{smallmatrix} a_1 \\ a_2 \end{smallmatrix} \right) = a_1v_1 + a_2v_2$, which is the orthogonal projection.

Answer 4

I found this approach helpful so I would like to share it.

Useful results, if $P$ is a projection matrix then:

1. $ P\circ P = P$ , i.e projector $P$ projects the kernel onto itself.
2. $ Pv=v\ \forall\ v\in V$ where $V=\mathrm{null}(P)$, direct consequence of $1.$
3. $ Pu=\emptyset\ \forall\ (u \mid u.v=\emptyset\ \forall v\in V) $
4. $ P^{\dagger}=P $, iff $P$ represents an orthogonal projection.

using $(2)$ it is clear that $V$ is spanned by the eigenvectors of P which correspond to eigenvalue $\lambda=1$. Let $w1,w2$ be the eigenvectors for $\lambda=1$, we know that $W=(w1,w2)$ would diagonalize the matrix $P$. Thus, $W^TPW=1$ or $P=WW^T$ this is what one would call an orthonormal basis.

Now we want to change the basis from $W$ to $V$ giving $W=S^{-1}V$ and $S=VV^T$ which would give you a good starting point. One way to see it is to notice that $(V^TV)^{-1}$ is a kind of normalization factor.