The physical distinguishing feature for the special subgroups of the matrix groups $U(n)$, $O(n)$ etc seems to be that the handedness of the coordinate system does not change if the transformation belongs to a special subgroup. I have thought about it but cannot make any connection or headway to prove it. How would I go about characterizing this handedness to show that it is left unchanged under such transformations?
The answer to your question depends on your understanding of "handedness".
The easy "abstract nonsense" answer goes as follows: Two bases $(e_1,\ldots,e_n)$ of the given $n$-dimensional real vector space $V$ are called equivalent if the unique linear transformation $T:\ V\to V$ defined by $Te_i:= f_i$ $(1\leq i\leq n)$ has positive determinant. It is easy to see that there are exactly two equivalence classes. One says that two bases in the same class have the same handedness. Since transformations $T\in SO(n)$ have determinant $1$ they preserve the handedness of any system of $n$ linearly independent vectors $x_1$, $\ldots$, $x_n$.
But of course you desire a geometrical intuitive or "physical" explanation of this equivalence relation. Now this is a more complicated story. The set of all bases $(e_1,\ldots,e_n)$ of $V$ is an $n^2$-dimensional space $X$ in its own right, and it turns out that $X$ consists of exactly two disjoint components. Two bases in the same component can be continuously deformed into each other so that at each time $t$ between $0$ and $1$ we have $n$ linearly independent vectors. (The proof of this fact is too long for this answer.) This means that we have a continuous map $t\mapsto T_t\in GL(n)$ with $T_0(e_i)=e_i$ and $T_1(e_i)=f_i$ $(1\leq i\leq n)$.
But for two bases not in the same component such a deformation is not possible. To prove the latter statement one has to look at $\det(T_t)$ as a function of $t$. This determinant is $=1$ at $t=0$ and $<0$ at $t=1$, so it has to vanish for some $t$ in between.
Those transformations that preserve the handedness are simply those with a positive determinant. One can easily see if two transformations have a positive determinant so does their product and inverses, hence they form a subgroup.