about $\operatorname{SO}(2)$ group When 
$$
A =\begin{pmatrix} a & b \\ c & d \end{pmatrix},\quad  a,b,c,d \in \mathbb{R}, \quad A^2 -2aA + I = 0
$$
Is $ A \in \operatorname{SO}(2)\; $ if $A^n \in SO(2)\;$ for some $n \in \Bbb N$?
$\operatorname{SO}(2)$ is special orthogonal group.
 A: I'm a bit surprised since nobody tried to find a counter-example! Let $p$ be a nonzero real number and
$$A = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & -p \\ p^{-1} & 1 \end{pmatrix}.$$
Then we have $A^2 - \sqrt{2}A + I = O$ and $A^4 = -I \in SO(2)$, but still $A \notin SO(2)$ unless $p = \pm 1$.
But this is partially true if some auxiliary conditions are imposed. For example, let's denote $a = \cos \theta$, where $\theta$ is allowed to have complex value in the case $|a| > 1$. Also, let $n$ be the least positive integer satisfying $A^n \in SO(2)$. If we impose the condition that $\sin n\theta \neq 0$, then $A \in SO(2)$. (The condition $\sin n\theta \neq 0$ corresponds to the condition that $A^n \neq \pm I$.)
Let's prove this claim. Here we further assume that $a \neq \pm 1$, so that $\sin \theta \neq 0$. Later this excluded cases will be treated separately. Since
$$x^2 - 2ax + 1 = (x - e^{i\theta})(x - e^{-i\theta}),$$
we have
$$x^n = (x^2 - 2ax + 1)q_n (x) + \frac{\sin n\theta}{\sin\theta} x - \frac{\sin (n-1)\theta}{\sin \theta}.$$
Now by assumption and the fact pointed out by Michael Banaszek, we can write
$$\frac{\sin n\theta}{\sin\theta} A - \frac{\sin (n-1)\theta}{\sin \theta} I = A^n = R_{\varphi}, \ \cdots \ (1)$$
where
$$R_{\varphi} = \begin{pmatrix} \cos \varphi & -\sin\varphi \\ \sin\varphi & \cos \varphi \end{pmatrix}$$
is the rotation matrix corresponding to the angle $\varphi$. Now comparing the trace of both sides of (1), we find that
$$\sin n\theta \cos \theta - \sin(n-1)\theta = \sin \theta \cos \varphi,$$
or by addition formula for sine, we have
$$\cos\varphi = \cos n\theta.$$
In particular, $\theta$ is real and $\theta = \frac{1}{n}(\varphi + 2\pi k )$ for some $k \in \mathbb{Z}$. Plugging this back to (1) and simplifying, we obtain
$$\quad A = R_{\theta}.$$
Here we critically used the fact that $\sin n\theta = \sin \varphi$ is nonzero.
Now we treat the case when $a = \pm 1$. In this case, we hvae
$$ x^n = (x^2 - 2ax + 1)q_n (x) + n a^{n-1} x - (n-1) a^{n} .$$
Then by assumption,
$$ n a^{n-1} A - (n-1) a^{n} I = A^n = R_{\varphi}$$
for some $\varphi$. Now take $k$-power to both sides to obtain
$$ kna^{kn-1}A - (kn-1)a^{kn}I = A^{kn} = R_{k\varphi}.$$
Dividing both sides by $kn a^{nk-1}$, we have
$$ A = a \left( 1 - \frac{1}{kn}\right)I + \frac{a^{1-nk}}{nk} R_{k\varphi}.$$
But since $a^{1-nk} R_{k\varphi}$ is uniformly bounded, taking $k \to \infty$ gives
$$ A = aI,$$
which is again in $SO(2)$.
A: The answer is yes I believe. Your requirement that $A^2-2aA+I=0$ forces $\det(A)=1$, so we know that if $A^n \in SO(2)$, then $\det(A^n)=\det(A)=1$, so we've eliminated the problem that Qiaochu mentioned. Now, it can be shown using a little bit of algebra that any matrix $B$ in $SO(2)$ is of the form $$\begin{pmatrix} \sin x & -\cos x \\ \cos x & \sin x \end{pmatrix}\ $$ Looking at a few powers of this matrix, we see that $$ B^2= \begin{pmatrix} -\cos 2x & -\sin 2x \\ \sin 2x & -\cos 2x \end{pmatrix}\ \text{and  } B^3= \begin{pmatrix} -\sin 3x & \cos 3x \\ -\cos 3x & -\sin 3x \end{pmatrix}\ $$ Thus, if we have $A^n \in SO(2)$, then if we finagle the signs and possibly switch $\sin $ and $\cos $ in a very natural way, it seems like $A$ will also be of the proper form with angle $\frac{x}{n}$, implying $A \in SO(2)$.
This isn't a rigorous proof, but I believe it at least gives the method of it.
Edit: As Shiyu pointed out, if $B$ is instead written in the form $$ \begin{pmatrix} \cos x & -\sin x \\ \sin x & \cos x \end{pmatrix}\ $$ then $$ B^k = \begin{pmatrix} \cos kx & -\sin kx \\ \sin kx & \cos kx \end{pmatrix}\ $$ so if $A^n \in SO(2)$ then $$ A^n = \begin{pmatrix} \cos y & -\sin y \\ \sin y & \cos y \end{pmatrix}\ \implies A= \begin{pmatrix} \cos \frac{y}{n} & -\sin \frac{y}{n} \\ \sin \frac{y}{n} & \cos \frac{y}{n} \end{pmatrix}\ $$ so $A \in SO(2)$
A: If $A^n=I$ for $n=2,3,...$, I think the answer is yest.
$A\in SO(2)$ if and only if $\det A=1$ and $AA^T=I$.
Since $A^2(A^2)^T=I$ and $A^3(A^3)^T=I$, we have $A^3(A^3)^T=AA^2(A^2)^TA^T=AA^T=I$. In addition, as Michael Banaszek mentioned $\det A=1$, we conclude $A\in SO(2)$.
A: I think the answer is not necessarily. Because the orthogonal group is generated by reflections (and, for SO(n) , the largest order is n, i.e., if R is a rotation in SO(n), R is the composition of at-most n reflections). Take, then, a reflection Re with $Re^2=Ro$ , where $Ro$ is a rotation . Then $Re^{2n}=(Re^2)^n$ is a rotation, as a
composition of rotations, but Re itself is not a rotation.
A similar argument for reflections Re' with $Re^{2n+1}=Ro'$ shows that if $A^n$ is a rotation, it does not follow that A must be a rotation.
