A question about irreducible representations of SO(3) group I have a perplexity about the representations of the group of 3D rotations $SO(3)$.
I'm reading the following text which is describing the irreducible representations of $SO(3)$ that are associated to so called Wigner-D Matrices.


'..any matrix representation $\textbf{D}(g)$ of $SO(3)$ that acts on some vector space $V$..'

Can the group $SO(3)$ act on a more general vector space $V$ that is not $\mathbb{R}^3$? To me this sounds weird. I would describe a representation of $SO(3)$ as a map
$$\rho : SO(3) \rightarrow GL_3(\mathbb{R})$$
So for each fixed $g \in SO(3)$,   $\rho(g):\mathbb{R}^3 \rightarrow \mathbb{R}^3$ describes a rotation matrix.
In the document above it is stated that the irreps of $SO(3)$ are indexed by  a $l \ge 0$ such that any rotation (a general representation that I've described above) can be decomposed as a direct sum of representations acting on suitable vector spaces of dimension $2l+1$. If we take $l=2$ does it make any sense to consider $\rho(g)^{(5)}:\mathbb{R}^5 \rightarrow \mathbb{R}^5$ ?
I mean, the group is about 3D rotations...  Maybe I'm just pretty confused about it.
 A: Yes, $SO(3,\Bbb R)$ is the naturally seen as the group of all (linear) rotations of $\Bbb R^3$. However, it acts on other spaces (I mean, other than $\Bbb R^3$). For instance, it acts on the space $\Bbb R^{3\times3}$ of all real $3\times3$ matrices this way: if $M\in SO(3,\Bbb R)$ and $A\in\Bbb R^{3\times3}$, then$$M.A=MAM^{-1}.$$But $\Bbb R^{3\times3}$ is a $9$-dimensional vector space.
A: I wanted to explore José's answer in the context of the passage of the text book.  In other words, believing the textbook, what does it say about José's example?
I'll write $V$ as a short hand for the collection of $3\times 3$ matrices.  As José noted, this this is a $9$-dimensional vector space on which $SO(3)$ acts, so it really comes from a map $SO(3)\rightarrow Gl(V)\cong Gl_9(\mathbb{R})$.
Now, the first thing to note is that conjugation by things in $SO(3)$ fixes the identity $I\in V$ (because conjugation by any invertible matrix does).  Moreover, conjugation fixes all real multiples of the identity.
This means that $V$ has a decomposition as $V\cong V_0 \times V'$ where $V'$ is some $8$-dimensional representation of $SO(3)$ and where $V_0 = \operatorname{span}\{I\}$.  By the way, this must be the $V_0$ of your text because a one-dimensional vector space has no non-trivial decomposition at all.
But we can actually identify $V'$. To do so, consider the trace of a matrix.  Note that if $A\in V$ and $B\in SO(3)$, then $tr(BAB^{-1}) = tr(A)$.  Thus, the $SO(3)$ action also preserves the subspace of $V$ consisting of matrices with trace $0$.  Since $V_0$ only intersects this subspace at the $0$-matrix, we may thus identify $V'$ with the traceless $3\times 3$ matrices.
Since $\dim V' = 8$ is even, it must further decompose.  The key to this decomposition is the fact that conjugating by $SO(3)$ preserves symmetric and antisymmetric matrices separately.  That is, if $A\in V$ with $A^t = \lambda A$ with $\lambda \in \{\pm 1\}$, then for any $B\in SO(3)$, we have $$(BAB^{-1})^t = (BAB^t)^t = BA^t B^t = B\lambda A B^t = \lambda BAB^{-1}.$$
Now, the antisymmetric matrices form a subspace $V_1$ of $V'$ of dimension $2(1)+1 = 3$, while the symmetric matrices form a subspace $V_2$ of $V'$ of dimension $2(2) + 1 = 5$.  Since the only matrix which is both symmetric and antisymmetric is the $0$ matrix, we have a decomposition $V'\cong V_1\oplus V_2$.
I claim that both $V_1$ and $V_2$ are irreducible, so that, in particular, the $V_1$ and $V_2$ in the book you are reading.
The proof that $V_1$ is irreducible is easy.  Based on the text you're reading, it if decomposed, it would have to be decompose as a sum of $3$ copies of $V_0$.  Since the action of $SO(3)$ on $V_0$ is trivial, it would then follow that the action of $SO(3)$ on $V_1$ is trivial.  But it's easy to verify that this is not the case.  For example, taking $B = \operatorname{diag}(-1,-1,1)\in SO(3)$ and $A = \begin{bmatrix} 0 & 0 & 1\\ 0 & 0 & 0\\ -1 & 0 & 0\end{bmatrix}$, then $BAB^{-1} = -A\neq A$.
For $V_2$, we have to work a bit harder to show that it is irreducible.  If it is irreducible, then by your text, it decomposes either as a sum of 5 copies of $V_0$ or as $V_1\oplus V_0\oplus V_0$.  To show that neither of these occur, it's enough to show that there is no copy of $V_0$ in $V_2$.  That is, that there is no subspace of $V_2$ which is fixed by everything in $SO(3)$.
A general element of $V_5$ looks like $\begin{bmatrix} a & b & c\\ b & d & e \\ c & e & f\end{bmatrix}$ with $a+b+f = 0$.  Conjugating by the three diagonal matrices in $SO(3)$ quickly shows that the only way such an element can be fixed by $SO(3)$ is if it is diagonal.  Now, conjugating by $\begin{bmatrix} 0 & 1 & 0\\ 1 & 0 & 0\\ 0 & 0 & -1\end{bmatrix}\in SO(3)$ now shows that $a=d$.  Similarly, one finds that $d =f$.  Since $a=b=f$ and $a+b+f = 0$, we have now shown that the only vector in $V_2$ which is fixed by all of $SO(3)$ is the $0$-matrix.
Thus, $V_2$ cannot contain a copy of $V_0$, so it must be irreducible.
In summary, José's example decomposes as $V\cong V_0 \oplus V_1\oplus V_2$.
A: To get an understanding of the higher-dimensional representations, think of the $SO(3)$ action as a rotation of functions instead of a rotation of vectors. This is how we get higher-dimensional representations, since one can have a large number of linearly independent functions.
For the irreducible representations of $SO(3)$ of dimension $2l+1$, the basis functions are the spherical harmonics with parameters $l, m$, where $m=-l,...,l$.
To get a concrete picture, someone visualised this action for $l$ up to 12 here
