# How to calculate "undo" rotation except for parallel rotation

Let's say we start with a vector $$u = [0, 0, 1]$$ which represents the positive z-axis.

Now, let's say we arbitrarily transform this vector via a rotation $$z' = R \begin{bmatrix}0 \\ 0\\ 1 \end{bmatrix}$$.

There are arbitrarily many rotations that can lead to the same $$z'$$.

I want to calculate a new rotation from $$R$$, $$R_{undo}$$, which inverts $$R$$ except it does not invert any portion of the rotation corresponding to a planar rotation around $$z'$$ (e.g. $$z'$$ is the normal vector of the plane of rotation).

For example, let's say in Euler angles with order zyx, $$R = [90, 90, 0]$$. Then, using the same zyx convention, $$R_{undo} = [0, -90, 0]$$, not $$[-90, -90, 0]$$.

How do we calculate $$R_{undo}$$ in general, for an arbitrary axis u?

It seems to me that your $$R_{\text{undo}}$$ is not very well defined, and that we should assume everything is in $$\mathbb{R}^3$$. From your example, if $$z'=Ru$$ for some rotation $$R$$, it looks like you want $$R_{\text{undo}}$$ to be the unique rotation that maps $$z'\mapsto u$$ and fixes the plane spanned by the vectors $$z'$$ and $$u$$ (take a moment to think about what this means geometrically). But note that if $$z'=-u$$, it is not at all clear what $$R_{\text{undo}}$$ should do -- the vectors won't span a plane, and there is no "prefered direction" to bring $$z'$$ back to $$u$$ anymore.
So let's put this problem aside, and assume $$z'\neq-u$$. Then you can calculate $$R_{\text{undo}}$$ following the procedures in the answers to this question, which looks very much like yours: