Calculate the matrix for the projection of $R^3$ onto the plane $x+y+z=0$. Calculate the matrix for the projection of $R^3$ onto the plane $x+y+z=0$.
If $b=\begin{bmatrix} x \\ y \\ z \\ \end{bmatrix}$ and $A=\begin{bmatrix} 1 & 0 \\ 0 & 1 \\ -1 & -1 \\ \end{bmatrix}$ I get stuck when trying to solve using $$A^TA\hat x=A^Tb$$ and $$proj_vb=A\hat x$$
$A^T=\begin{bmatrix} 1 & 0 & -1 \\ 0 & 1 & -1 \\ \end{bmatrix}$   
therefore $A^TA=\begin{bmatrix} 2 & 1 \\ 1 & 2 \\ \end{bmatrix}$  
and $A^Tb=\begin{bmatrix} x-z \\ y-z \\ \end{bmatrix}$  
So $$\begin{bmatrix}2 & 1 \\ 1 & 2 \\ \end{bmatrix} \hat x = \begin{bmatrix} x-z \\ y-z \\ \end{bmatrix}$$
Let's say $\hat x=\begin{bmatrix} a \\ b \\ \end{bmatrix}$,
then $2a+b=x-z$ and $a+2b=y-z$  
Here is where I get confused, what is $\hat x$ so that I can plug it into $proj_vb=A\hat x$ and get the projection vector?
 A: Orthogonal Projection from a basis
The matrix for a Projection satisfies
$$
A^2=A
$$
However, for an Orthogonal Projection, we must also have
$$
A=A^T
$$
Since $\frac1{\sqrt2}(1,0,-1)$ and $\frac1{\sqrt2}(1,0,-1)\times\frac1{\sqrt3}(1,1,1)=\frac1{\sqrt6}(1,-2,1)$ form an orthonormal basis for the space so that $x+y+z=0$, we get that
$$
\begin{bmatrix}
\frac1{\sqrt2}&\frac1{\sqrt6}\\
0&-\frac2{\sqrt6}\\
-\frac1{\sqrt2}&\frac1{\sqrt6}
\end{bmatrix}
\begin{bmatrix}
\frac1{\sqrt2}&0&-\frac1{\sqrt2}\\
\frac1{\sqrt6}&-\frac2{\sqrt6}&\frac1{\sqrt6}
\end{bmatrix}
=
\begin{bmatrix}
\frac23&-\frac13&-\frac13\\
-\frac13&\frac23&-\frac13\\
-\frac13&-\frac13&\frac23
\end{bmatrix}
$$
is the projection onto the space so that $x+y+z=0$.

Orthogonal Projection from a unit normal
We can also use Jyrki Lahtonen's approach and use the unit normal $\frac1{\sqrt3}(1,1,1)$ to get
$$
\begin{bmatrix}
1&0&0\\0&1&0\\0&0&1
\end{bmatrix}
-
\begin{bmatrix}
\frac1{\sqrt3}\\\frac1{\sqrt3}\\\frac1{\sqrt3}
\end{bmatrix}
\begin{bmatrix}
\frac1{\sqrt3}&\frac1{\sqrt3}&\frac1{\sqrt3}
\end{bmatrix}
=
\begin{bmatrix}
\frac23&-\frac13&-\frac13\\
-\frac13&\frac23&-\frac13\\
-\frac13&-\frac13&\frac23
\end{bmatrix}
$$

General Projection
For any vector $v$ so that $v\cdot(1,1,1)\ne0$, we have that
$$
Ax=x-\frac{x\cdot(1,1,1)}{v\cdot(1,1,1)}v
$$
is a projection onto the space where $x\cdot(1,1,1)=0$. This projection only depends on the direction of $v$, not the length. Thus, there is a two dimensional family of projections, parameterized by a vector in $\mathbb{P}^2$. The matrix for this projection is
$$
\begin{align}
A
&=\begin{bmatrix}
1&0&0\\0&1&0\\0&0&1
\end{bmatrix}
-
\frac
{v
\begin{bmatrix}
1&1&1
\end{bmatrix}}
{v\cdot
\begin{bmatrix}
1&1&1
\end{bmatrix}}\\[6pt]
&=
\begin{bmatrix}
1-\frac{v_1}{v_1+v_2+v_3}&-\frac{v_1}{v_1+v_2+v_3}&-\frac{v_1}{v_1+v_2+v_3}\\
-\frac{v_2}{v_1+v_2+v_3}&1-\frac{v_2}{v_1+v_2+v_3}&-\frac{v_2}{v_1+v_2+v_3}\\
-\frac{v_3}{v_1+v_2+v_3}&-\frac{v_3}{v_1+v_2+v_3}&1-\frac{v_3}{v_1+v_2+v_3}\\
\end{bmatrix}
\end{align}
$$
As matrices, vectors are represented here by column vectors. Note that when $v=(1,1,1)$, we get the orthogonal projection mentioned above in the first two sections.
A: Here's how to do it for any plane $V$ through the origin. Let $\vec{n}$ be a vector normal to $V$. Then the orthogonal projection of a vector $\vec{u}=(x,y,z)$ is
$$
P_V(\vec{u})=\vec{u}-\frac{\vec{u}\cdot\vec{n}}{||\vec{n}||^2}\vec{n}.
$$
Why? Check that i) $P_V$ is linear, ii) $P_V(\vec{u})=\vec{u}$ whenever $\vec{u}\perp \vec{n}$, and iii) $P_V(\vec{n})=\vec{0}$ 
(add whatever properties your definition of an orthogonal projection needs to have).
Then all you need to do to get the matrix of $P_V$ w.r.t. the basis $\{\mathbf{i},\mathbf{j},\mathbf{k}\}$ is to calculate their images. With the above formula that is easy!
A: You defined $\hat{x} = \begin{bmatrix}a \\ b\end{bmatrix}$, so you just have to figure out $a$ and $b$ to find out what $\hat{x}$ is.
