Why is the orthogonal complement actually orthogonal to the projection? How do you show that $q=x-p$ is orthogonal to the projection ($p$)? I can understand it intuitively from drawing it out. But can't show it's true with proof.
What I know is we will have to show that the dot product of $x-p$ and $p$ is $0$. And if I can do that I can show that the angle between them is $90$ degrees. I just don't know how to show this algebraically? Please help. This is the definition for vector projection that I was given:
Let $x$ and $y$ be vectors in $ \mathbb{R}^2$ (or in $ \mathbb{R}^3$ ) with $y \ne 0$. Let $u = \frac{1}{||y||} y$
be the unit vector in the direction of y. Then we define the vector projection of x onto y is $p=\alpha u = \frac{x^Ty}{y^Ty} y$, where $\alpha$ is the scalar projection which is defined as $\alpha = \frac{x^Ty}{||y||} $
Nowhere in the definition does it directly say that projection is perpendicular.
I'm think if I can somehow show that $p^Tq=0 $ then I would have solved the problem. I tried substituting in $(x-p)$ in for $q$ and I subbed in $\frac{x^Ty}{y^Ty} y$  for $ p$. I am now stuck.
 A: For the inner product I prefer the notation $(\vec x\cdot \vec y)$  instead of $x^Ty$. So, using the linearity of the inner product and the definition of projection
$$
\vec p=\frac{(\vec x\cdot\vec y)}{(\vec y\cdot\vec y)}\vec y
$$
we have
$$
((\vec x-\vec p)\cdot\vec p)=(\vec x \cdot \vec p)-(\vec p \cdot \vec p)=
$$
$$
\frac{(\vec x\cdot\vec y)}{(\vec y\cdot\vec y)}(\vec x\cdot\vec y)-\frac{(\vec x\cdot\vec y)}{(\vec y\cdot\vec y)}\frac{(\vec x\cdot\vec y)}{(\vec y\cdot\vec y)}(\vec y\cdot\vec y)= 
$$
$$
=\frac{(\vec x\cdot\vec y)^2}{(\vec y\cdot\vec y)}-\frac{(\vec x\cdot\vec y)^2}{(\vec y\cdot\vec y)}=0
$$
A: So using that the projection matrix onto $v$ is $P=uu^T$ with $u$ a unit vector in the direction of $v$ and we will prove that $Px$ is orthgonal to $x-Px$ directly by calculating $(Px)^T(x-Px)$. To do this we will need to use that $P=P^T$ and $P^2=P$ which are easily verified from the formula given.
We can see that $$(Px)^T(x-Px) = x^TP^T(x-Px)=x^TP^Tx-xP^TPx \\=x^TPx -x^TP^2x=x^TPx-x^TPx=0$$ and therefore $Px$ is orthogonal to $x-Px$.
A: Using your notation, with $p^Tq$ as the inner product,
if you substituted $x - p$ for $q$ and substituted $\frac{x^Ty}{y^Ty} y$ for $p$
you should have gotten this:
$$ \frac{x^Ty}{y^Ty} y^T \left(x - \frac{x^Ty}{y^Ty} y \right). $$
Linear properties of the inner product imply that if $u,$ $v,$ and $w$ are vectors and $\lambda$ is a scalar, then
$$ u^T (v - \lambda w) = u^T v - \lambda u^T w.$$
Keeping in mind that $\frac{x^Ty}{y^Ty}$ is a scalar, you can apply this fact to
$y^T \left(x - \frac{x^Ty}{y^Ty} y \right),$ after which you should see some further simplifications that you can make.
