Linear operator and rotation matrices I have come across a question involving a linear operator $A$ that is represented by the following matrix:
$$
\begin{pmatrix}
0 & 1 & 0\\
1 & 0 & 0 \\
0 & 0 & 2
\end{pmatrix}
$$
This matrix represents the operator in $\mathbb{R^3}$. Now if I need to find the matrix representation after a rotation around the z-axis by 90 degrees, I would have thought that the matrix would simply be $AR$ where A is the matrix above and $R$ is the matrix representing the rotation. However, the solution I was given was that the linear transformation $A'$ is given by $$ A' = R A R^{-1}$$. What is the explanation behind this?
 A: $AR$ rotates a vector around the z-axis by $90^\circ$ and then applies the linear transformation $A$. However, that's not what they're asking for. They are asking for the representation of $A$ in a coordinate system that has been rotated by $90^\circ$.
Now, without the rotation, understanding $A$ is rather simple: It turns $x$ into $Ax$. However, with the rotation, all of the vectors are now rotated around by $90^\circ$. In terms of $R$, this means all of the vectors which were $z$ before the rotation are now $Rz$. Therefore, the representation of $A$ with this rotated coordinate system can be thought of as a transformation from $Rx$ to $RAx$.
If we refer to this representation $A$ as $A'$, then we have:
$$A'(Rx)=RAx \text{ for all } x\in \Bbb{R}^3$$
Since this equation holds for all $x\in \Bbb{R}^3$, the matrices on both the left and right must be equal transformations, so we get:
$$A'R=RA\rightarrow A'=RAR^{-1}$$
Therefore, the representation of $A$ in a coordinate system that has been transformed by $R$ is $RAR^{-1}$.
A: If a linear operator $A$ maps each vector from $\Bbb R^n$ to $\Bbb R^n$ and is invertible, then a rotation applied on $A$ resulting a new operator $A'$, should map the rotated vectors in $\Bbb R^n$ to rotated counterparts in $\Bbb R^n$, say, if $$u=Av\quad,\quad u,v\in \Bbb R^n$$then for some rotation matrix $R$ we should have $$Ru=A'(Rv)$$which means that if $A$ maps $v$ to $u$ bijectively, then similarly $A$ maps $Rv$ to $Ru$ bijectively. The above equation means that $$u=R^{-1}A'Rv=Av$$therefore $$R^{-1}A'R=A$$or equivalently $$A'=RAR^{-1}$$
A: I like to view this sort of thing in terms of input and output bases: the matrix of a linear transformation $T$ eats coordinate tuples of vectors that are expressed relative to some basis $\mathcal B$ and spits out coordinate tuples in some, possibly different, basis $\mathcal B'$. As pointed out by others, you’re not being asked to compute the composition of a rotation and the given transformation, but the express that transformation relative to a basis obtained by rotating the standard basis.  
To put the task a slightly different way, you need to find a matrix that will eat and spit out vectors expressed relative to the rotated basis. You have a matrix that eats and spits out vectors in the standard basis, so a way to accomplish this is to first convert the vector to the standard basis, apply the matrix that you have, and then convert its output to the rotated basis. If $R$ is the matrix that converts coordinates from the standard basis to the rotated basis, then it should be clear that $R^{-1}$ performs the conversion in the other direction. Putting this all together, you get $RAR^{-1}$.  
(You didn’t ask why its $RAR^{-1}$ instead of $R^{-1}AR$, which can also be a bit confusing. That’s covered in many places: look for discussions of passive vs. active transformations.)
