How to remove roll from axis angle rotation I have an object in 3d space oriented along the global axes, with Z axis pointing forward, Y axis pointing up and X axis pointing right. I need to reorient this object to face a certain vector target t.
I can use axis angle rotation to calculate the axis and angle needed to face that target. The axis would be the cross product between normalized initial direction vector the object was pointing at, and the normalized direction vector towards the target. The angle is the inverse cosine of their dot product. I can then create 4x4 matrix M of that angle axis rotation and assign that matrix to the object.
The matrix M we calculated may consist of rotations along all three global axes. So our object may have yaw, pitch and roll in its new orientation. Now let's say the object is a head. When we rotate our heads to point towards certain direction we try to avoid rolling it. So we rotate it (mostly) only along two axes. What would the best way to copy that behaviour, i.e. remove roll from when tracking a target.
One solution I see is to instead of calculating angle axis rotation, we create an orientation matrix with three orthogonal vectors. Where forward vector is pointing towards the target, the Up vector is the cross product of normalized forward vector and the global Right vector. And we can use the global right vector as our Right vector. That way we avoid any roll.
But how would I remove roll directly from axis angle rotation? Do I negate the roll after applying the rotation? Specifically by rotating the object around its new forward axis,  to make the object's right axis orthogonal with the global Up axis?
Or what is generally the best approach to do what I need to do?
 A: Your two solutions do not appear to match each other in general.
The second method ("rotating the object around its new forward axis, to make the object's right axis orthogonal with the global Up axis") is consistent with yaw-pitch-roll systems that I have seen elsewhere, for example the one illustrated in
this figure.
Unless we have non-zero roll, the transverse axis of the object
(your "right" vector) stays in the horizontal plane (your global $XZ$ plane),
that is, orthogonal to your global "up" axis.
By rotating your object until its "right" is orthogonal to global "up", you achieve this version of zero roll.
But using the second method you end up with a "right" vector that is not usually the same as the global "right" vector.
Your first method cannot work quite the way you described it.
In most cases, presumably the vector to your target object is not in the global $YZ$ plane and not orthogonal to the global "right" vector.
So if you point your new "forward" vector at the target, but continue to use the global "right" vector as your body "right" vector, you now have orientation vectors that are not orthogonal.
You can obtain a set of orthogonal vectors by putting your new body "right" vector orthogonal to your new "forward" and "up" vectors (which you can do by using the cross product of those vectors). In most cases this will give you a vector that is not orthogonal to the global "up" vector, so this method is not equivalent to your second method.
A method that gives the same result as the second method is to point the new "forward" vector at the target, construct a new "right" vector orthogonal to the new "forward" vector and the global "up" vector, and then construct a new "up" vector orthogonal to the new "forward" and "right" vectors.
If you really must have yaw and pitch angles (that is, if the orientation matrix is not enough), 
what you get from your second method could be considered as a form of spherical coordinates, with the spherical axis identified with the global "up" axis,
the pitch angle identified with latitude, and the yaw angle identified with longitude. So you could use any good method of converting Cartesian coordinates to spherical coordinates in order to convert the Cartesian $XYZ$ coordinates of your target into the yaw and pitch angles necessary to face that target.
One way to convert the Cartesian coordinates to angles is to set the yaw to
$ \mathrm{atan2}(X,Z) $ radians
and the pitch to
$ \mathrm{atan2}(Y,\sqrt{X^2+Z^2}) $ radians,
where $\mathrm{atan2}(v,u)$ is the two-parameter arc tangent function offered by many software math libraries
(for example this).
Note that a positive yaw then represents a clockwise rotation as seen from above;
if you want a counterclockwise rotation angle, use $\mathrm{atan2}(-X,Z).$
A positive pitch angle corresponds to a target above the horizontal global plane.
