Calculating $\underset{R \in \text{SO}(3) \\ Re_3 = Qe_3}{\arg\max} \text{Tr}(R)$ for fixed $Q \in \text{SO}(3)$ In a problem that I'm currently working on, I have a term of the form $\underset{R \in \text{SO}(3) \\ Re_3 = Qe_3}{\arg\max} \text{Tr}(R)$ for some fixed $Q \in \text{SO}(3)$. I was hoping to find a closed form solution to this problem (say, $R$ in terms of $Q$). I also would like to avoid Euler angle solutions, as they will significantly complicate the problem that I'm working on, but if it is the only way, it would be better than nothing. As of now, I really have no ideas where to begin. I know that we can write $\text{Tr}(R) = 1 + 2\cos(\theta)$, where $\theta$ is the angle of rotation of $R$. I thought maybe the constraint $Re_3 = Qe_3$ would tell me something about their axes of rotation, from which the Rodrigues formula could be used. Any help is appreciated.
 A: I assume that $e_3$ refers to the third standard basis vector, $(0,0,1)$.
It seems that the nature of the solution depends nontrivially on whether $Qe_3$ is a multiple of $e_3$. I will address the "generic" case in which $Qe_3$ is not a multiple of $e_3$.
We will construct an orthonormal basis $v_1,v_2,v_3$ of $\Bbb R^3$. Let $v_1 = e_3$.  By the Gram Schmidt process, we can find a unit vector $v_2$ such that $v_1 \perp v_2$ and $Qe_3 = av_1 + bv_2$ (for some $a,b \in \Bbb R$). By changing the sign (of all entries of) $v_2$ if necessary, we cn ensure that $a \geq 0$. Finally, let $v_3$ be a unit vector orthogonal to both $v_1,v_2$ (for instance, we could use the cross-product $v_1 \times v_2$).
Let $V$ denote the matrix with columns $v_1,v_2,v_3$. Let $S = V^TRV$; notably, for any $R \in SO(3)$, $S$ is an element of $SO(3)$ has the same trace as $R$. The statement that $R$ satisfies $Re_3 = Q e_3$ is equivalent to the statement that the first column of $S$ is given by $(a,b,0)$. In other words, we want to determine the matrix $S \in SO(3)$ of the form
$$
S = \pmatrix{a&*&*\\b&*&*\\0&*&*}
$$
with maximal trace.  From our previous specifications, $a,b$ are non-negative values with $a^2 + b^2 = 1$. It is easy to see that the maximum possible value of $S_{22}$ (subject to the constraint that the completed matrix $S$ is orthogonal) is given by $a$ (note that we adjusted our basis so that $a \geq 0$). Similarly, it is easy to see that the maximum possible value of $S_{33}$ (again, such that $S$ is orthogonal) is $1$. Thus, it must be the case that the completion
$$
S = \pmatrix{a&-b&0\\b&a&0\\0&0&1}
$$
has the largest possible trace, since all $3$ diagonal entries are as large as they can be given the problem constraints. The $R$ corresponding to this $S$ is given by $R = VSV^T$.
Note: If we go through the Gram-Schmidt process, we see that $a = \cos(\theta)$, where $\theta$ is the angle between the vectors $e_3$ and $Q e_3$.

I'll now try to translate this result into a convenient formula for $R$. We can write
$$
S = I + \pmatrix{a-1&-b&0\\b&a-1&0\\0&0&0}.
$$
We can use this to conclude that
$$
R = VSV^T = I + (a-1)[v_1v_1^T + v_2v_2^T] + b[v_2v_1^T - v_1v_2^T].
$$
Following the procedure used to attain these $a,b,v_1,v_2$, we have the following:
$$
\begin{align}
a &= |e_3^TQe_3| = |Q_{33}|\\
b &= \pm \sqrt{1 - a^2}\\
v_1 &= e_3\\
u &= Qe_3 - e_3e_3^TQe_3 = Q e_3 - Q_{33}e_3\\
v_2 &= \pm \frac{u}{\|u\|}
\end{align}
$$

The remaining cases where $Qe_3 = \pm e_3$, by the way, are easy. We can simply take
$$
R = \pmatrix{1&0&0\\0&1&0\\0&0&\pm 1}.
$$
A: The problem can be rephrased as maximising $\operatorname{trace}\left((Q^TR)Q\right)$ subject to $Q^TR\in SO(3)$ and $Q^TRe_3=e_3$. The constraints imply that $Q^TR=\pmatrix{X\\ &1}$ for some $X\in SO(2)$. If we partition $Q$ as $\pmatrix{A&\ast\\ \ast&q_{33}}$, we may further reformulate the problem as
$$
\text{maximising} \operatorname{trace}\left(XA\right)+q_{33}
\quad\text{subject to}\quad X\in SO(2).\tag{1}
$$
The solution to $(1)$ is well-known: if $USV^T$ be a singular value decomposition of $A$, a global maximiser is given by $X=V\operatorname{diag}(1,\det(UV^T))U^T$. Hence
$$
R=Q\pmatrix{V\operatorname{diag}(1,\det(VU^T))U^T\\ &1}
$$
is a global maximiser to the original problem and the maximum objective function value is $\sigma_1(A)+\sigma_2(A)\det(VU^T)+q_{33}$.
