Rotate XYZ frame in 3D space Given a XYZ frame in 3D space at origin O(0,0,0). And given a plane equation:
L = 0.5774x + 0.1155y + 0.8083z = 0.
I need to derive the matrix which rotate the XYZ frame until Z-axis is vertical to the plane L.
 A: Your plane is given by the equation
$$
L = 0.5774 x + 0.1155 y + 0.8083 z = 0 
$$
from this one can derive the normal vector
$$
n = \frac{(0.5774, 0.1155, 0.8083)^t}{|(0.5774, 0.1155, 0.8083)^t|}
= (0.5774, 0.1155, 0.8083)^t
$$
Thus $L$ was already given as
$$
L = n \cdot r = 0
$$
for a vector $r = (x,y,z)^t$.
Looking for a vector $u = (1, y, 0)^t \in L$:
$$
u \in L \iff 
n \cdot u = 
n_x + n_y y = 0 \iff 
y = -\frac{n_x}{n_y} = -5
\iff u = (1, -5, 0)^t \\
n_u = \frac{u}{|u|} = (0.19612,-0.98058, 0)^t 
$$ 
So we have the normal unit vector $n$ of the plane and a unit vector $n_u$ within the plane.
Looking for another unit vector $n_v \perp n_u \wedge n_v \perp n$:
$$
n_v = n_u \times n = \epsilon_{i,j,k} \, e_i \, n_{u j} \, n_k
= (-0.79260, -0.15852, 0.58884)^t
$$
That is a nice orthonormal system, with two vectors in the plane, and one perpendicular to it. 
The matrix
$$
R = (n_u, n_v, n)
$$
fullfills
$$
R e_x = n_u \quad R e_y = n_v \quad R e_z = n
$$
where $e_x = (1, 0, 0)^t$, $e_y = (0,1,0)^t$ and $e_z = (0, 0, 1)^t$.
However $\det(R) = -1$, which means this is not a proper rotation, therefore we switch to 
$$
R = (n_v, n_u, n) =
\left(
\begin{matrix}
-0.79260 &  0.19612 &  0.57740 \\
  -0.15852 & -0.98058 &  0.11550 \\
   0.58884 &  0.00000 &  0.80830
\end{matrix}
\right)
$$
A: You have single condition to have the Z axis oriented the same direction as the normal of the plane. Which means, that you'll have infinitely many possibilities.
You have the vector of $Z$ axis $\vec{z}$ and vector of the normal of the plane $\vec{n}$.
If you rotate the coordinate system around the vector $\vec{z} + \vec{n}$ by $\pi$, you'll have the Z axis to the original direction of normal vector.
I used that in OpenGL, where is a nice function glRotatef.
If you want some specific orientation of XY plane, you'll have to rotate again around the Z axis.
