Determining whether an orthogonal matrix represents a rotation or reflection The exercise asks us to determine whether the given orthogonal matrix represents a rotation or a reflection. If it is a rotation, give the angle of rotation; if it is a reflection, give the line of reflection.
$$
A = \begin{bmatrix}
       -\frac{3}{5} & -\frac{4}{5}\\[0.3em]
       -\frac{4}{5} & \frac{3}{5}\\[0.3em]
     \end{bmatrix}
$$
I know you can check whether it is a reflection or rotation by calculating the determinant. So for example for the matrix above
$$det(A) = -\frac{3}{5}\cdot\frac{3}{5}-(-\frac{4}{5})\cdot(-\frac{4}{5}) = -1$$
And so that means that matrix $A$ corresponds to a reflection in $R^2$, but how do you get the line of reflection from this?
$$
B = \begin{bmatrix}
       -\frac{1}{2} & \frac{\sqrt{3}}{2}\\[0.3em]
       -\frac{\sqrt{3}}{2} & -\frac{1}{2}\\[0.3em]
     \end{bmatrix}
$$
And for a rotation, so for example matrix $B$ given above, can you simply say that it corresponds to the rotation matrix $R$
$$ 
  R =   \begin{bmatrix}
       cos(\theta) & -sin(\theta)\\[0.3em]
       sin(\theta) & cos(\theta)\\[0.3em]
     \end{bmatrix}
$$
So this gives
$$cos(\theta) = -\frac{1}{2}, \theta = cos^{-1}(-\frac{1}{2}) = 120^{\circ}\\
  sin(\theta) = -\frac{\sqrt{3}}{2}, \theta = sin^{-1}(-\frac{\sqrt{3}}{2}) = -60^{\circ}
$$
Which means that matrix $B$ corresponds to a counterclockwise rotation of $120^{\circ}$, right?
 A: Here is a general guideline for $2 \times 2$ orthogonal matrices.
They have one of the two forms
$$\text{Either} \ \ R =   \begin{bmatrix}
       a &-b\\[0.3em]
       b &  \ \  \ a\\[0.3em]
     \end{bmatrix} \ \ \ \ \text{or} \ \ \ \ S =   \begin{bmatrix}
       a & \ \ \ b\\[0.3em]
       b & -a\\[0.3em]
     \end{bmatrix}$$
with norm $1$ column vectors (thus $a^2+b^2=1$), the first case with $\det(A)=a^2+b^2=1$, the second with  $\det(A)=-(a^2+b^2)=-1$. 
More precisely, they have the form (you have cited the first one, the second one is less known...):
$$R_{\theta} =   \begin{bmatrix}
       \cos(\theta) & -\sin(\theta)\\[0.3em]
       \sin(\theta) & \ \ \ \cos(\theta)\\[0.3em]
     \end{bmatrix} \ \ \ \ \ \ \text{or} \ \ \ \ \ \ S_{\alpha}=\begin{bmatrix}
       \cos(2 \alpha) & \ \ \ \sin(2 \alpha)\\[0.3em]
       \sin(2 \alpha) & -\cos(2 \alpha)\\[0.3em]
     \end{bmatrix} $$
where $\theta$ is the rotation angle, of course, and $\alpha$ is the polar angle of the axis or symmetry i.e., the angle of one of its directing vectors with the x-axis.
Thus, for your question, once you have recognized that a matrix is a symmetry matrix, it suffices to pick the upper left coefficient $ \cos(2 \alpha)$ and identify the possible $\alpha$s, with a disambiguation brought by the knowledge of $ \sin(2 \alpha)$.
A: Hint for the first question: If a vector is on the line of reflection, it is unchanged by that reflection.
For the second, $\sin \frac{2\pi}3 = \frac{\sqrt3}2$, so double-check that you’ve gotten all of the signs right.
A: for the line of reflection look for a mirror lines that stays fixed under the transformation. for example, in the case of $A = \begin{bmatrix}
       -\frac{3}{5} & -\frac{4}{5}\\[0.3em]
       -\frac{4}{5} & \frac{3}{5}\\[0.3em]
     \end{bmatrix}$  try the line $x = \pmatrix{5\\5a}.$  we want $Ax =  x$ that is 
$$\pmatrix{-3 -4a\\-4  + 3a } =  \pmatrix{5\\5a}$$ which gives you $a= -2.$  therefore the line $y = -2x$ is the mirror.
