Rotation matrix and projection matrix in $\Bbb R^3$ Task:
Write down the matrix of the linear transformation, which is performing the following: 


*

*rotates every vector to 45 degrees counterclockwise around OY 

*and then, project it onto a plane, which goes through points (0,0,0), (1,0,-1), (0,1,-1)
Solution that I found is:
$\begin{bmatrix} \sqrt{2}/2 & 0 & -\sqrt{2}/2 \\ 0 & 1 & 0  \\ \sqrt{2}/2 & 0 & \sqrt{2}/2 \end{bmatrix}$ - this is the rotation matrix
$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$ - this one projects vectors onto a plane
Am I correct at making such matrices? I have little experience at working with transformation matrices of any spaces other than $\Bbb R^2$
 A: This problem looks like it was designed to trap the unwary. If you rotate counterclockwise about the positive $y$-axis, the unit $z$ vector $(0,0,1)^T$ should get mapped to $(1/\sqrt2,0,1/\sqrt2)$, but the matrix you’ve got rotates in the opposite direction. The correct rotation matrix is the inverse of yours (which is also simply the transpose).  
Your projection matrix is the identity. Technically, it is a projection since it’s idempotent, but it doesn’t do anything at all, let alone project onto a plane. Perhaps you were thinking of starting with $\operatorname{diag}(1,1,0)$ and performing a change of basis. There are other ways to construct this projection, but the easiest is to compute the orthogonal projection matrix onto the normal to the plane, then subtract that from the identity matrix. This works because the orthogonal projection of a vector onto the plane is what’s left after you eliminate its component orthogonal to the plane.  
A basic formula you should learn for the orthogonal projection matrix onto a vector $\mathbf n$ is ${\mathbf n\mathbf n^T\over\mathbf n^T\mathbf n}$. In this case, you can take advantage of working in $\mathbb R^3$ and use a cross product to compute a vector normal to the given plane.
