Transformation matrix for triangle projection in $xy$ plane

It's been a long time since I did computer graphics and algebra, and I don't remember how to correctly manipulate linear transformations.

My scenario: I have 3 points, $a,b,c \in \mathbb{R}^3$, forming a triangle embedded in $\mathbb{R}^3$. I need a transformation (i.e., matrix), that correctly takes the points to $xy$-plane. So, formally with all specifications:

Find $a'$, $b'$, and $c'$ such that: the vector $a'b'$ lies in the $y$-axis and $a'$ must be the origin. All three points must lie in $xy$-plane, and $\|a'b'\| = \|ab\|$, $\|a'c'\| = \|ac\|$, and $\|b'c'\| = \|bc\|$, so that $\operatorname{Area}(abc) = \operatorname{Area}(a'b'c')$. Remember that $a$, $b$, and $c$ form a valid triangle. I need the projection of $\operatorname{Triangle}(a,b,c)$ given this constraints.

I know I need to do a couple rotations and translations and find a combination of then, but can't figure out if it is possible to put all the conditions in a single matrix.

-
Welcome to math.SE! I've edited your post a bit, hopefully making it a bit clearer, but if I accidentally changed what you wanted to say, I apologize, and please do feel free to fix it. – Zev Chonoles May 15 '12 at 23:44
You must, in general, have a translation. The 3 points define a hyperplane. Either it passes through the origin or not. If not, no invertible linear transformation will map it to a hyperplane through the origin. – copper.hat May 16 '12 at 2:08

Let $u$ and $v$ be vectors. Suppose the matrix $M$ is an isometry (preserves distances), takes $u$ to the $y$-axis and $v$ into the $x{-}y$ plane. Then it must take $u\times v$ to the $z$-axis and $u\times(u\times v)$ to the $x$-axis. That is $$M\begin{bmatrix}u&u\times v&u\times(u\times v)\end{bmatrix}=\begin{bmatrix}0&0&|u||u\times v|\\|u|&0&0\\0&|u\times v|&0\end{bmatrix}\tag{1}$$ So we get $$M=\begin{bmatrix}0&0&|u||u\times v|\\|u|&0&0\\0&|u\times v|&0\end{bmatrix}\begin{bmatrix}u&u\times v&u\times(u\times v)\end{bmatrix}^{-1}\tag{2}$$ Thus, $M$ takes three orthogonal vectors to three orthogonal vectors, preserving their lengths, so $M$ is an isometry.
If we let $u=b-a$ and $v=c-a$ and compute $M$ as in $(2)$, then $M(b-a)$ lies on the $y$-axis, and $M(c-a)$ lies in the $x{-}y$ plane. The only thing we can't do is map $a$ to the origin. A linear map sends $0$ to $0$, and since $M$ preserves distances $|Ma|=|M(a-0)|=|a-0|=|a|$. So unless $a=0$, $Ma\not=0$. To map $a$ to the origin, we need to apply a translation as well.
Compute $M$ as above, then the affine-linear map $A(x)=M(x-a)$ satisfies the requirements.