Distance between $3\times3$ matrices (isometries) The problem is pretty long, so I'm going to describe it while writting what I have so far.
First I have to asociate the $3\times3$ matrices with $\Bbb R^9$, so we defined $\phi:M_{3\times3}(\Bbb R) \to  \Bbb R^9$ as the following mapping: $(a_{ij}) \mapsto (a_{1,1},...,a_{9,9})$. Then define the distance between matrices (using the distance in $\Bbb R^9$), so we get $d: M_{3\times3}(\Bbb R) \times M_{3\times3}(\Bbb R) \to \Bbb R$ where $(A,B) \mapsto d_{\Bbb R^9}(\phi(A),\phi (B))$, where $d_{\Bbb R^9}$ is the usual metric of $\Bbb R^9$, clearly $d$ is a metric. 
Now, I have to prove that if we restrict the distance between the matrices to the elements of $O (3,\Bbb R)$,wich is the orthogonal group, for every $M \in SO(3,\Bbb R)$ exists $\varepsilon>0$ such that if $d(M,M')<\varepsilon$ then $M'$ is not a reflection.
I don't know how to do that, we know that the distance $d$ is a continuous function, and the determinant of the elements in $SO(3,\Bbb R)$ is allways $1$, but the determinant of a reflection is $-1$,I belive we have to use that, maybe compose the distance with the determinant, since the determinant is also continuous (isn't it?).
 A: In dimension 3, if $$ A A^t = I,  \; \; \det A = 1,  $$ and
$$ B B^t = I,  \; \; \det B = -1,  $$
then $$ 2 \leq \operatorname{dist} (A,B) \leq 2 \sqrt 3.   $$
In addition, if $A=I$ and $B$ is a genuine reflection, meaning no rotation in the (setwise) fixed plane and eigenvalues $-1,1,1,$ then the distance between $I(A)$ and $B$ is exactly $2.$ For this example, you could just have  $B$ the diagonal matrix with diagonal entries $-1,1,1.$
Let me explain that your distance is simply this: the distance of any matrix $M$ from the matrix with all entries $0$ is just $\sqrt {\operatorname{trace} M M^t}.$ See FROBENIUS.    Very important that you know $$ \operatorname{trace} M N^t =  \operatorname{trace} M^t N =  \operatorname{trace}  N M^t =  \operatorname{trace}  N^t M.  $$
Then for any pair of matrices, 
$$  \operatorname{dist} (M,N) = \sqrt {\operatorname{trace} (M - N) (M^t - N^t)}.   $$
Oh, if we have some $R R^T = I,$ then
$$  \operatorname{dist} (RM,RN) =  \operatorname{dist} (MR,NR) =   \operatorname{dist} (M,N). $$ For this one, we do not care about the determinant of $R,$ it is allowed $\pm 1.$
As a result, given $A A^t = 1, \det A = 1,$ while $B$ is a genuine reflection, then
$$   \operatorname{dist} (A,AB) =   \operatorname{dist} (A,BA)  $$ is exactly $2.$
On the theory that this does little harm if nobody reads it; the dot product on $\mathbb R^{n^2}$ is exactly this thing I wrote before, 
$$ \operatorname{trace} M N^t =  \operatorname{trace} M^t N =  \operatorname{trace}  N M^t =  \operatorname{trace}  N^t M.  $$
Write it out for 2 by 2 matrices $M,N,$ you'll see.  
EDIT, Wednesday, why not. As in comment, we are finding the minimum of $   \operatorname{dist} (I,C)$ where $\det C = -1, \; \; C C^T = I.$ Writing
$$ d =  \operatorname{dist} (I,C) $$ we get $$ d^2 = \operatorname{trace} (I - C) (I - C^t) = \operatorname{trace} (2I - C - C^t)  = \operatorname{trace} (2I - 2C),   $$
or $$ d^2 = 2 \operatorname{trace} (I - C).   $$
We are in dimension 3, which is odd. The eigenvalues of $C$ are $-1, \cos \theta + i \sin \theta,  \cos \theta - i \sin \theta.$ As a result, the trace of $C$ is $2 \cos \theta - 1,$ so $-3 \leq \operatorname{trace} C \leq 1,   $ then $-1 \leq \operatorname{trace} (-C) \leq 3,   $ next $2 \leq \operatorname{trace} (I-C) \leq 6,   $
finally $$4 \leq 2 \operatorname{trace} (I-C) \leq 12.   $$ With $ d =  \operatorname{dist} (I,C) $ and $ d^2 = 2 \operatorname{trace} (I - C),   $ we arrive at 
$$4 \leq  d^2 \leq 12,   $$ so $2 \leq d  \leq 2 \sqrt 3,   $ that is
$$2 \leq \operatorname{dist} (I,C) \leq 2 \sqrt 3,   $$ 
for dimension 3, $\det C = -1, \; C C^T = I.$
NOTES, Thursday, 12 September 2013:
$$\begin{array}{l}\text{What is  reflection, saith my sufferings, then?}\cr
\text{If all the pens that ever poets held}\cr
\text{Had fed the feeling of their masters' thoughts,}\cr
\text{And every sweetness that inspir'd their hearts,}\cr
\text{Their minds, and muses on admired themes;}\cr
\text{If all the heavenly quintessence they still}\cr
\text{From their immortal flowers of poesy,}\cr
\text{Wherein, as in a mirror, we perceive}\cr
\text{The highest reaches of a human wit;}\cr
\text{If these had made one poem's period,}\cr
\text{And all combin'd in beauty's worthiness,}\cr
\text{Yet should there hover in their restless heads}\cr
\text{One thought, one grace, one wonder, at the least,}\cr
\text{Which into words no virtue can digest.}\end{array}$$
A reflection in $\mathbb R^3$ negates one vector, call it $r,$ (and demand that $r$ be unit length) but fixes every vector in the plane orthogonal to $r.$ We can take an orthonormal basis of the plane, call that $s,t.$ Each of these is fixed. So $r,s,t$ are an orthonormal basis of eigenvectors (of the reflection) with eigenvalues $-1,1,1.$ That is to say, in this basis, the matrix of the reflection is
the diagonal matrix
$$ D \; = \;  
 \left(  \begin{array}{rrr}
  -1  &  0  &  0 \\
  0   &  1  &  0 \\
  0  &  0   &  1  
\end{array} 
  \right).
$$
Thus, if $P$ is the matrix with comlumns given by the entries $r,s,t,$ then $P P^T = I.$ If the matrix of the reflection in the original basis is $W,$ then $$ D = P^T W P  $$ and
$$ W = P D P^T.  $$
So, what do we know about $W?$ That it is evidently symmetric, orthogonally similar to $D,$ that $W^2 = I,$ and finally $\operatorname{trace} W = 1. $ For dimension $n$ instead of dimension $3,$ the definition of reflection would be $W$ symmetric, $W^2 = I,$ and $\operatorname{trace} W = n-2.$ 
Finally, as I proved above in dimension $3,$ the reflections are the closest orientation-reversing isometries to the identity matrix, distance $2.$
A: Yes, your approach is correct. Let $\epsilon=1$ and you're about done.
