Rotate 3d plane I have a plane in 3D space that formed from 3 poin $P_1=(x_1, y_1, z_1)$, $P_2=(x_2, y_2, z_2)$, $P_3=(x_3, y_3, z_3)$
I want to rotate and transform this points (equally related plane) into 2D space (Avoiding $z$ axis but save distance and relations in 2D plane).
 A: As a first step, I'd move $P_1$ to the origin, so that the points become $P_1=(0,0,0)$, $P_2=(x_2-x_1, y_2-y_1, z_2-z_1)$, $P_3=(x_3-x_1, y_3-y_1, z_3-z_1)$.  From there it's just a question of applying a rotation matrix.
A rotation matrix $\mathbf{R}$ which preserves all distances and shapes etc is
given by
$$
\mathbf{R}=\left(
\begin{array}
[c]{ccc}%
e_{1}^{x} & e_{2}^{x} & e_{3}^{x}\\
e_{1}^{y} & e_{2}^{y} & e_{3}^{y}\\
e_{1}^{z} & e_{2}^{z} & e_{3}^{z}%
\end{array}
\right)
$$
where the vectors $\left(  e_{1}^{\alpha},e_{2}^{\alpha},e_{3}^{\alpha
}\right)  $ for $\alpha\in\left\{  x,y,z\right\}  $ are orthogonal and of unit
length, i.e.
$$
\sum_{i=1}^{3}e_{i}^{\alpha}e_{i}^{\beta}=\left\{
\begin{array}
[c]{ccc}%
1 &  & \alpha=\beta\\
0 &  & \alpha\neq\beta
\end{array}
\right. .%
$$
Applying $\mathbf{R}$, your rotated points $P_{i}^{\prime}=\left(
x_{i}^{\prime},y_{i}^{\prime},z_{i}^{\prime}\right)  $ are given by
$$
\left(
\begin{array}
[c]{c}%
x_{i}^{\prime}\\
y_{i}^{\prime}\\
z_{i}^{\prime}%
\end{array}
\right)  =\mathbf{R}\left(
\begin{array}
[c]{c}%
x_{i}\\
y_{i}\\
z_{i}%
\end{array}
\right)
$$
I suggest making the top row of $\mathbf{R}$ the unit
vector in the direction from $P_{1}$ to $P_{2}$, i.e.
$$
\left(
\begin{array}
[c]{c}%
e_{1}^{x}\\
e_{2}^{x}\\
e_{3}^{x}%
\end{array}
\right)  =\frac{1}{\sqrt{\left(  x_{1}-x_{2}\right)  ^{2}+\left(  y_{1}%
-y_{2}\right)  ^{2}+\left(  z_{1}-z_{2}\right)  ^{2}}}\left(
\begin{array}
[c]{c}%
x_{1}-x_{2}\\
y_{1}-y_{2}\\
z_{1}-z_{2}%
\end{array}
\right)
$$
which would mean that the line from $P_{1}$ to $P_{2}$ gets mapped into the
$x$-axis.
Working out what you could use for the second and third rows of $\mathbf{R}$ is now just a question of solving some simple linear equations, to make sure that the line from $P_{1}$ to $P_{3}$ has no z component.  
To solve for $\left( e_{1}^{z},e_{2}^{z},e_{3}^{z}\right)$, the vector
$e_{i}^{z}$ must have a zero dot-product with both $e_{i}^{x}$ and $\left(
x_{3}-x_{1},y_{3}-y_{1},z_{3}-z_{1}\right)  $, so the equations are
$$
\begin{align*}
e_{1}^{x}e_{1}^{z}+e_{2}^{x}e_{2}^{z}+e_{3}^{x}e_{3}^{z}  & =0\\
\left(  x_{3}-x_{1}\right)  e_{1}^{z}+\left(  y_{3}-y_{1}\right)  e_{2}%
^{z}+\left(  z_{3}-z_{1}\right)  e_{3}^{z}  & =0
\end{align*}
$$
Hence
$$
\left(
\begin{array}
[c]{c}%
e_{1}^{z}\\
e_{2}^{z}\\
e_{3}^{z}%
\end{array}
\right)  =\lambda_{z}\left(
\begin{array}
[c]{c}%
\left(  z_{3}-z_{1}\right)  e_{2}^{x}-\left(  y_{3}-y_{1}\right)  e_{3}^{x}\\
\left(  x_{3}-x_{1}\right)  e_{3}^{x}-\left(  z_{3}-z_{1}\right)  e_{1}^{x}\\
\left(  y_{3}-y_{1}\right)  e_{1}^{x}-\left(  x_{3}-x_{1}\right)  e_{2}^{x}%
\end{array}
\right)
$$
for some $\lambda_z$, which should be determined so that $e_{i}^{z}$ is a vector
of unit length.  Finally $e_{i}^{y}$ can be determined as the vector which is
perpendicular to both $e_{i}^{x}$ and $e_{i}^{z}$, i.e.
$$
\left(
\begin{array}
[c]{c}%
e_{1}^{y}\\
e_{2}^{y}\\
e_{3}^{y}%
\end{array}
\right)  =\lambda_{y}\left(
\begin{array}
[c]{c}%
e_{3}^{z}e_{2}^{x}-e_{2}^{z}e_{3}^{x}\\
e_{1}^{z}e_{3}^{x}-e_{3}^{z}e_{1}^{x}\\
e_{2}^{z}e_{1}^{x}-e_{1}^{z}e_{2}^{x}%
\end{array}
\right)
$$
where again $\lambda_{y}$ is determined so that $e_{i}^{y}$ is a vector of
unit length.
A: As the accepted answer suggests, first apply a translation of $-P_1$ to
all points.  Your three points are then $O,$ $A,$ and $B$ (where
$A=P_2-P_1$ and $B=P_3-P_1$).  Now compute the plane unit normal:
$$
  n = \frac{A \times B}{| A\times B|}.
$$
To rotate the plane to the $xy$ plane, all you need to do is rotate 
$n$ to $(0,0,1)$.  The simplest way to do this is to rotate $n$ about
the $z$ axis into the positive $x$ half of the $xz$ plane and then 
rotate about the $y$ axis until $n$ is parallel to the positive $z$ axis.
You could also use Rodrigues' rotation formula to rotate about the axis
$n\times (0,0,1),$ but this rotation matrix is much more complicated
than the matrix for a rotation about a coordinate axis.
Computing the rotation matrix that rotates $n$ about the $z$ axis into the positive $x$ half of the $xz$ plane does not require computing an angle.  The rotation matrix is simply given in terms of $n$:
$$
  R_z = 
  \left(
  \begin{array}{lll}
  n_x/\sqrt{n_x^2+n_y^2} & n_y/\sqrt{n_x^2+n_y^2} & 0\\
  -n_y/\sqrt{n_x^2+n_y^2} & n_x/\sqrt{n_x^2+n_y^2} & 0\\
  0 & 0 & 1
  \end{array}
  \right).
$$
The rotation matrix that rotates $n'=R_zn$ about the $y$ axis to $(0,0,1)$ also does not require computing an angle.  It is 
$$
  R_y = 
  \left(
  \begin{array}{lll}
  n'_z & 0 & -n'_x \\
  0 & 1 & 0\\
  n'_x & 0 & n'_z
  \end{array}
  \right).
$$
A: You sound like you could benefit from learning how to represent direction in 3D using a direction vector.  This entails learning how to normalize a vector down to unit length, which results in a direction vector.
Then, once comfy with that, learn how to solve for the orientation of (direction perpendicular to) the plane of a 3D triangle:
  o <-- normalize[(P2-P1) x (P3-p2)]   (vector cross product of 2 vector diffs)

Now, you want to define the desired 3D coordinate rotation which aligns the orientation of your triangle-plane with the z-axis.  Again, using cross products,
these are the 3 columns of the 3x3 rotation matrix you want:
R =  [ newXaxis    newYaxis   newZaxis ]  (three orthogonal direction vectors)
By choice, you choose your value of o as newZaxis.
The other two new axes have to be orthogonal to newZaxis and to each other.
The normalized vector cross product gives you the mutual orthogonal direction to
any two distinct directions d1 an d2 (so long as they are not coincident or opposite directions):
newXaxis =  normalize(newZaxis x [ 1 0 0 ] )
(if newZaxis == [ 1 0 0 ], use newXaxis =  normalize(newZaxis x [ 0 1 0 ] ) instead
newYaxis = newZaxis x newYaxis 
Now, coordinate rotate each point of your triangle by rotator R: 
p1' <-- R • p1   (vector times a matrix)
p2' <-- R • p2
p3' <-- R • p3
You'll notice that p1', p2' and p3' all have identical z-coords.  You can throw these out, and just use their x and y coordinates.  You've effectively projected your triangle into a 2D x-y plane.   
