A property of orthogonal matrices Let $R$ be a $3\times 3$  orthogonal matrix. Let $v$ be the unit vector such that $Rv=v$ (upto sign change). Consider any unit vector $u$ such that $u^{T}v=0$ where $T$ stands for transpose. Show that $u^{T}Ru = \frac12(trace(R)-1)$.
The problem arises in finding the angle of rotation given a rotation-transformation matrix. The vector $v$ would be the axis of rotation since it is invariant under the transformation $R$. The angle of rotation can be obtained by observing how much $u$ rotated by. That is, $\cos(\theta)=u^TRu$ where $\theta$ is the rotation angle. I'm not able to understand how the trace of $R$ enters the formula.  
 A: Analyze  Rodrigues formula for rotation.
Let it be in a such form: $R(v,\theta)=I+\sin(\theta)S(v)+(1-\cos(\theta))S^2(v)$.   
The expression $\sin(\theta)S(v)$ is the skew-symmetrical part of rotation $R$,
   $I+(1-\cos(\theta))S^2(v)$ is its symmetrical part. 
Trace of any skew-symmetrical matrix is equal $0$ so $\text{trace}(R)=\text{trace}(\text{sym(R)})$.
Additionally $S^2(v)=vv^T-I$, where $\Vert v \Vert=1$, -    notice that in this case we have also $\text{trace}(vv^T)=1$.
Now calculating $\text{trace}(R)$ we obtain  
$\text{trace}(R) =\text{trace}  (I)+\text{trace}((1-\cos(\theta))(vv^T-I))=3-2(1-\cos(\theta))$.
Hence $\text{trace}(R)=1+2 \cos(\theta) $ 

It can be also other way of proving it.
Rotation matrix $R$ could be transformed to the other basis where axis of rotation would be z-axis.
Such operation doesn't affect the trace 
(algebraically it is expressed as $R(v,\theta)=R_{Trans}R(z,\theta)R_{Trans}^{-1}$ ).     
For rotation about z-axis we have $R(z,\theta)=\begin{bmatrix}\cos(\theta) &   -\sin (\theta)   & 0   \\
\sin (\theta)  & \cos (\theta)   & 0   \\
0  & 0  & 1  \end{bmatrix}$.
From it $\text{trace}(R)=1+2 \cos(\theta) $ follows.
A: You did not say explicitly, but I am assuming your orthogonal matrix is real. The norm of the eigenvalues are 1 for orthogonal matrices and complex eigenvalues for real matrices come in conjugate pairs. Combining these two facts with $Rv=v$ (eigenvalue 1, no sign change!), we have that the eigenvalues for $R$ are: ${e^{i\theta},e^{-i\theta},1}$. Then
$$
\frac{1}{2}(trace(R)-1)=\frac{1}{2}(e^{i\theta}+e^{-i\theta}+1-1)=\cos(\theta)=u^TRu.
$$
