distance SO(3) rotation matrix According to M. Moakher's

Means and averaging in the group of rotations

and I. Sharf's

Arithmetic and geometric solutions for average rigid-body rotation

the distance between two rotation matrices is $$\| R_1 - R_2 \|_{\text{F}}$$ where $\| \cdot \|_{\text{F}}$ denotes the Frobenious norm. Does it mean $\left\| R_1 - R_2 \right\|_{\text{F}}$ or $\left\|  R_1^T R_2 \right\|_{\text{F}}$?

*

*$\left\| R_1 - R_2 \right\|_{\text{F}}$ does not make sense as $(R_1-R_2) \notin$ SO(3).


*$\left\|  R_1^T R_2 \right\|_{\text{F}}$ is also strange: suppose $R_1=R_2$, the distance is $3$. I find out that  the distance between two rotation matrices is less than $3$. Such metric is against my intuition (the distance between two identical elements is largest!).
 A: The slightly counterintuitive bit is that the metric comes from the (normed) vector space structure of the matrix algebra, but when you're dealing with $SO_3$ you're focusing on the complementary structure (the bilinear operation of multiplication).  Still, the Frobenius norm gives us an extrinsic metric on $SO_3$ when we consider it embedded in matrix land:  $d(R_1, R_2) = ||R_1 - R_2||_F$.  $SO_3$ is not a metric vector space, but a subset of a metric is still a metric space.   
Let's look at another related example.
Consider the complex numbers, $\mathbb{C}$, which form an algebra over the reals under the operations of multiplication and addition.  We have a standard norm in the complex numbers, $$|a + bi| = \sqrt{a^2 + b^2}$$ which induces a metric in which the distance between $u = a+bi$ and $v = c + di$ is $d(u,v) = |u-v|$.  Now this metric gives a perfectly good way to measure the distance between any 
two complex numbers.  This metric still provides a consistent way to conceive of distance between elements if we restrict our attention to the unit circle (those complex numbers $z$ such that $|z| = 1$, which form a group under multiplication).  However, you might have the objection that, considering the unit circle as embedded in $\mathbb{C}$, this measure of distance is extrinsic.  If that's an issue for you, you can establish an intrinsic metric by taking the distance of two points on the unit circle to be the minimum length among all paths between the points restricted to the unit circle.  
Your matrix case is much the same.  True you are considering $SO_3$ which is a special subset of matrices that forms a group under multiplication, but that does not invalidate the Frobenius norm on all matrices as a measure of distance. 
A: It means $\|(R_1-R_2)\|_F.$
It does not matter that $R_1-R_2$ is not special-orthogonal as we can still take the norm. Consider that the distance between two points on a circle is related to the length of the vector between them, even though that vector is not itself on the circle. This is much like the $SO(2)$ version of your question.
A: In fact, this topic opens an interesting realm of ideas so I will include here a short discussion inspired by Rotation Averaging, Hartley 2013. As others mentioned, what you are looking for is simply $\|R_1-R_2\|_F$, a quantity that vanishes in the Frobenius sense as the two rotations approach each other. $\|R_1R_2^\top\|_F$ will always be constant as the rotations form a group, composition of rotations is a rotation, i.e., an orthogonal matrix whose Frobenius norm is $1$.
Let me discuss this a little. In the literature, the distance you mentioned is known as the chordal distance, $d_{\mathrm{chord}}$. It can also be understood as the Euclidean distance between the embeddings of two rotations into $\mathbb{R}^{3\times 3}$. It can be shown that:
\begin{align}
d_{\mathrm{chord}}(R_1,R_2) &= \|R_1-R_2\|^2_F = \|R_1 R_2^\top-I\|_F^2\\
&= 2\left(\sin^2(\theta) + (1-\cos(\theta))^2\right)\\
&= 2\sqrt{2} \sin(\theta/2)
\end{align}
where $\theta=d_{\langle}(R_1,R_2)$ is the angular distance of two rotations. In fact, $d_{\langle}$ is the (natural) geodesic metric such that $0\leq\theta\leq\pi$. In most applications, minimizing the squared chordal distance is preferred:
$$
d_{\mathrm{chord}}(R_1,R_2)^2 = 8 \sin^2(\theta/2). 
$$
For a set of rotations $\{R_i\in SO(3)\}_{i}$, the averages defined on chordal metrics are known as chordal averages:
$$
\bar{R} = \arg\min_{R} \,\, C(R)\triangleq\sum\limits_{i=1}^n d_{\mathrm{chord}}(R_i,R)^2
$$
$C(R) $is strictly convex in an open ball of radius at most $\pi/4$.
One advantage of employing chordal distances for averaging is the existence of certain known simple algorithms naturally minimizing cost functions involving chordal distances (e.g. in closed form). The caveat is that the chordal distance metric is not convex beyond a ball of radius $\pi/2$. The squared chordal distance has substantially different convexity properties compared to the squared geodesic distance.
