How to find representation of O(2) by its action on complex basis vectors I am trying to solve the following problem: 
I am unsure how to solve this. What I have tried is:


*

*I know the matrices (rotations+reflections) of O(2) in the Cartesian (x,y) basis. For example the matrices for the rotation around the z axis look like:
\begin{align}
R_{\theta}=
\begin{pmatrix}
\cos \theta &-\sin \theta\\
\sin\theta & \cos\theta\\
\end{pmatrix}
\end{align}
So, what I have thought is to try and transform the basis from the Cartesian to the spherical basis:
\begin{align}
(x^{'},y^{'})^{T}=(x+iy,x-iy)^{T}=
\begin{pmatrix}
1&i\\
1&-i\\
\end{pmatrix}
(x,y)^{T}
\end{align}
Call the matrix that does the conversion between the bases S and then in order to find $R_{\theta}$ in the new spherical basis, perform a similarity transformation, namely:
\begin{align}
R_{\theta}^{'}=S^{-1}R_{\theta}S
\end{align}
Where $R_{\theta}^{'}$ is the matrix in the spherical basis. 

*I have also tried acting with the matrix $R_{\theta}$ on the spherical basis, but that does not really lead to anything useful.


I suspect that what they are asking for is actually the homomorphism between O(2) and U(1) but I am unsure how I should go about establishing this.
Thanks
 A: If you represent a point $P(x,y)$ on the $xy$-plane as a 2-component column then the proper rotations in a plane are represented by $2\times 2$ orthogonal matrices with $\det=+1$ because they keep the distance between two points fixed i.e.\begin{equation}(x'^2+y'^2)=(x^2+y^2)\end{equation} These matrices form a group, called $SO(2)$. The most general $2\times 2$ matrix belonging to SO(2) can be parameterized as:
\begin{equation}R(\theta)=\begin{pmatrix}\cos\theta & \sin\theta\\-\sin\theta & \cos\theta\end{pmatrix}\end{equation} This is called the defining representation of SO(2). The most general element of $O(2)$ multiplies this by the reflection matrix \begin{equation}M_x=\begin{pmatrix}1 & 0\\0 & -1\end{pmatrix}\end{equation} Or \begin{equation}M_y=\begin{pmatrix}-1 & 0\\0 & 1\end{pmatrix}\end{equation} This is the fundamental representation of $SO(2)$ but it is reducible since it is an abelian group. However the following $3\times 3$ representation acting on 3-D real vector space
\begin{equation}\begin{pmatrix}A_x'\\ A_y'\\ A'_z\end{pmatrix}=\begin{pmatrix}\cos\theta & \sin\theta & 0\\-\sin\theta & \cos\theta & 0\\0 & 0 & 1 \end{pmatrix}\begin{pmatrix}A_x\\ A_y\\ A_z\end{pmatrix}\end{equation} is reducible.\
Consider the combinations $(A_x\pm iA_y)$. Then, \begin{equation}A_x'\pm iA_y'=(A_x\cos\theta-A_y\sin\theta)\pm i(A_x\sin\theta+A_y\cos\theta)\end{equation} $$=(A_x\pm iA_y)e^{\pm i \theta}$$ Thus the symmetric combination goes to the symmetric combination under rotation and the anti-symmetric combination goes to the anti-symmetric combination. Therefore, $$3=1\oplus1\oplus1$$ for $(A_x+ iA_y)$, $(A_x- iA_y)$ and $A_z$. To show this explicitly, let us multiply the columns by \begin{equation}S=\begin{pmatrix}\frac{1}{\sqrt 2} & \frac{i}{\sqrt 2} & 0\\\frac{1}{\sqrt 2} & \frac{-i}{\sqrt 2} & 0\\0 & 0 & 1\end{pmatrix}\end{equation} and replace  $R(\theta)\rightarrow SR(\theta)S^{-1}$, $\vec A\rightarrow S\vec A$ and $\vec A'\rightarrow S\vec A'$.
\begin{equation}\begin{pmatrix}A_x'+iA'_y\\A'_x-iA'_y\\A'_z\end{pmatrix}
=\begin{pmatrix}e^{i\theta} & 0 & 0\\0 & e^{-i\theta} & 0\\0 & 0 & 1 \end{pmatrix}
\begin{pmatrix} A_x+iA_y\\ A_x-iA_y\\ A_z \end{pmatrix}\end{equation} This is a concrete example where a similarity transformation is used to perform a change of basis and how a reducible representation is be shown to be equal to the direct sum of the 1-dimensional irreducible representations. This is true for abelian groups.
