# Smallest rotation angle between quaternions accounting for symmetry

I am trying to compute a similarity measure between the orientation of 3D objects accounting for symmetry invariance. I have a set of 3D objects, which are defined by a center of mass, and 3 unit vectors associated to the orientation of this object such that $$vec_1$$ defines the orientation of the main axis, $$vec_2$$ the orientation of the second axis and $$vec_3$$ the orientation of the third axis.

I would like to compare the objects orientations by comparing their reference frame defined by these vectors. Quaternions seemed to be a nice way to do so, and I saw already several questions about it which helped me getting started.

In order to measure the relative rotation between two orientations, I first define the rotations matrices using the orientation vectors sorted by importance

$$R_{obj_i} = \begin{bmatrix}vec_{1_i} \\ vec_{2_i} \\ vec_{3_i}\end{bmatrix}$$

and I derive the corresponding "reference" quaternions $$q_{obj_i}$$ of each object. I found that one way to measure the relative angle $$\theta$$ between two quaternions $$q_{obj_i}$$ and $$q_{obj_j}$$ is to compute $$cos(\theta) = 2⟨q_{obj_i},q_{obj_j}⟩^2−1$$ where $$⟨q_{obj_i},q_{obj_j}⟩$$ denotes the dot product between the two quaternions. However this method does not account for the fact that several similar orientations have distinct quaternions. In my case, I could define 3 others quaternions using the rotation matrices:

$$\begin{bmatrix}-vec_{1_i} \\ -vec_{2_i} \\ vec_{3_i}\end{bmatrix},\begin{bmatrix}-vec_{1_i} \\ vec_{2_i} \\ -vec_{3_i}\end{bmatrix} and \begin{bmatrix}vec_{1_i} \\ -vec_{2_i} \\ -vec_{3_i}\end{bmatrix}$$

which should be considered similar as the first one. In order to account for this, I changed the measure of the relative angle $$\theta$$ to be $$cos(\theta) = arg_{k}max\left( 2⟨p_k\times q_{obj_i},q_{obj_j}⟩^2−1 \right)$$

where $$p_k$$ are the quaternions defined with the coordinates (1,0,0,0), (0,1,0,0), (0,0,1,0) and (0,0,0,1).

My problems are:

1. I am unsure about my method. Is there simpler way to perform this kind of analysis ?
2. Using only an uniform random distribution of quaternions, I found that $$Average(cos(\theta)) \approx$$ 0.2357. I have been trying to figure out where this comes from using this formalism (hyperspherical coordinates). In the simplest case, not accounting for any symmetry, $$Average(cos(\theta))$$ should be $$Average(cos(\theta)) = \frac{1}{2\pi^2}\int_0^\pi\int_0^\pi\int_0^{2\pi} cos(2\alpha)*sin^2(\alpha)sin(\beta)\ d\gamma\ d\beta\ d\alpha = -0.5$$ I have been trying to reduce the 4-space to obtain 0.235, however I an not sure that this is the right way to proceed.

Any help would be really appreciated.