Why does a space of symmetric traceless tensors form an irreducible representation of $SO(3)$? I have asked a similar question on Physics StackExchange, but did not get an answer. In the chapter IV.1 "Reducible or Irreducible?" of Zee's Group Theory book (p. 188-), the author breaks a 2nd rank tensor $T^{ij}$ into invariant subspaces with respect to the action of $\mathrm{SO(3)}$ group. The tensor $T^{ij}$ breaks into a five-dimensional (symmetric traceless), three-dimensional (antisymmetric) and one-dimensional invariant subspaces. Zee claims this five-dimensional space to be irreducible, i.e. it is implied that it does not have non-trivial invariant subspaces. Unfortunately, there is no proof.
Can someone explain to me why symmetric traceless tensors form an irreducible representation $\mathrm{SO(3)}$? References to relevant literature would be helpful as well.
 A: The "theory of highest-weight vectors" definitively answers many questions about finite-dimensional representations of (reductive, e.g....) compact real Lie groups, up to some fooling around with connectedness.
Namely, the complexified Lie algebra of the real Lie algebra of a compact real Lie group is (provably) reductive (often semi-simple), so has a Cartan ("diagonalizable") subalgebra $\mathfrak h$, and (with a choice) positive and negative root spaces $\mathfrak g_\alpha$ for roots $\alpha$.
In the finite-dimensional situation, a representation $V$ decomposes into simultaneous eigenspaces ("weight spaces") $V_\lambda$ for $\mathfrak h$ with $\lambda$ a linear functional ("weight") on $\mathfrak h$, and the action of the root spaces moves things around in an intelligible fashion. In particular, one proves that for irreducible $V$ there is a unique "weight" $\lambda$ such that $V_\lambda$ (is non-zero and) is annihilated by the root-space $\mathfrak g_\alpha$ for all positive roots $\alpha$, and this $V_\lambda$ is one-dimensional.
And conversely.
So in small situations we can look for highest-weight vectors, and if we find exactly one (up to scalar multiples), then the repn is irreducible.
A: Denote $S^{Tr}$ to be the space of symmetric traceless tensors.  Let $D\subset S^{Tr}$ be a non-trivial subspace of $S^{Tr}$ that is irreducible with respect to $SO(3)$.  We will prove  $D =  S^{Tr}$.
$D$ is closed with respect to $SO(3)$ transformations. Specifically:
$\forall \; T^{ab} \in   D, \; R_{x}^{a} \in SO(3)$, we have $R_{x}^{a}T^{xy}R_{y}^{b}\in D$.
If $T^{ab}\in D$ then $T^{ab}\in S^{Tr}$.  Also, if $\;T_{1}^{ab}, T_{2}^{ab}\dots  T_{n}^{ab} \in D$ then $\text{span}\{T_{1}^{ab}, T_{2}^{ab}\dots  T_{n}^{ab}\} \subset D$.
We must show that $\forall T\in S^{Tr}$ with $T\neq 0\;$, $\exists R_{1},R_{2} \dots R_{n}\in SO(3)$ such that  $\text{span}\{R_{1x}^{a}T^{xy}R_{1y}^{b}, R_{2x}^{a}T^{xy}R_{2y}^{b}\dots  R_{nx}^{a}T^{xy}R_{ny}^{b}\} = S^{Tr}$. In short, any non-zero tensor of $S^{Tr}$ can generate a basis for $S^{Tr}$ under the action of $SO(3)$, thus proving the ireductibility .
From now it's a problem of linear algebra. A symmetric traceless tensor $T^{ab}$ can be bought in the form  $\left(\begin{smallmatrix}
a & 0 & 0\\
0 & b & 0\\
0 & 0 & -(a+b)
\end{smallmatrix}\right)$ by a suitable rotation matix (symetric matrices are diagnoalizable). With the rotation matrix $\left(\begin{smallmatrix}
0 & 1 & 0\\
0 & 0 & 1\\
1 & 0 & 0
\end{smallmatrix}\right)$ we bring it to the form  $\left(\begin{smallmatrix} b & 0 & 0\\
0 & -(a+b) & 0\\
0 & 0 & a  
\end{smallmatrix}\right)$, with which we form a basis for all traceless diagonal tensors. By linear combinations we obtain the matrix $\left(\begin{smallmatrix} 1 & 0 & 0\\
0 & -1 & 0\\
0 & 0 & 0  
\end{smallmatrix}\right)$, which can be transformed via suitable rotation matrices in  $\left(\begin{smallmatrix} 0 & 1 & 0\\
1 & 0 & 0\\
0 & 0 & 0  
\end{smallmatrix}\right)$,$\left(\begin{smallmatrix} 0 & 0 & 1\\
0 & 0 & 0\\
1 & 0 & 0  
\end{smallmatrix}\right)$,$\left(\begin{smallmatrix} 0 & 0 & 0\\
0 & 0 & 1\\
0 & 1 & 0  
\end{smallmatrix}\right)$.
Starting with any traceceless symetric tensor, and applying only $SO(3)$ transformations and linear combinations, we can form a basis for $S^{Tr}$, thus obtain any possible traceceless symmetric tensor. Thus $S^{Tr}$ is irreducible with respect to $SO(3)$.
