I have an object in 3d space oriented along the global axes, with Z axis pointing forward, Y axis pointing up and X axis pointing right. I need to reorient this object to face a certain vector target
I can use axis angle rotation to calculate the axis and angle needed to face that target. The axis would be the cross product between normalized initial direction vector the object was pointing at, and the normalized direction vector towards the target. The angle is the inverse cosine of their dot product. I can then create 4x4 matrix
M of that angle axis rotation and assign that matrix to the object.
M we calculated may consist of rotations along all three global axes. So our object may have yaw, pitch and roll in its new orientation. Now let's say the object is a head. When we rotate our heads to point towards certain direction we try to avoid rolling it. So we rotate it (mostly) only along two axes. What would the best way to copy that behaviour, i.e. remove roll from when tracking a target.
One solution I see is to instead of calculating angle axis rotation, we create an orientation matrix with three orthogonal vectors. Where forward vector is pointing towards the target, the Up vector is the cross product of normalized forward vector and the global Right vector. And we can use the global right vector as our Right vector. That way we avoid any roll.
But how would I remove roll directly from axis angle rotation? Do I negate the roll after applying the rotation? Specifically by rotating the object around its new forward axis, to make the object's right axis orthogonal with the global Up axis?
Or what is generally the best approach to do what I need to do?