How many triples of mutually conjugate linear maps are there? I’m looking for finite-dimensional complex vector spaces $V$ with three invertible linear maps $r,s,t:V\to V$, satisfying the following “mutual-conjugation” condition:

*

*$rsr^{-1}=t$

*$sts^{-1}=r$

*$trt^{-1}=s$
For lack of a better term, I’m calling them conj-representations.
In order to avoid redundancies, say:

*

*Two conj-reps $V, V’$ are isomorphic if there is a linear isomorphism $f$ between them respecting the actions of the three maps (i.e. $fr=r’f$).

*A conj-rep is irreducible if it has no subconj-rep (i.e. the three maps do not restrict to a proper subspace).


Question: how many isomorphism classes of irreducible conj-reps are there?


Example: take $V=\mathrm{Span}(e_1,e_2,e_3)$ and:

*

*$r$ permutes $e_1 \leftrightarrow e_2$ and fixes $e_3$.

*$s$ permutes $e_2 \leftrightarrow e_3$ and fixes $e_1$.

*$t$ permutes $e_3 \leftrightarrow e_1$ and fixes $e_2$.

Note that this is not irreducible, as it has subconj-reps $\mathrm{Span}(e_1+e_2+e_3)$ and $\mathrm{Span}(e_1-e_2,e_2-e_3)$.
Another $1$-dimensional example: $r, s, t$ all act as $-1$.
I can prove that if all three maps are involutions, then these are all the conj-reps. I’m looking for examples where the maps are not involutions.
Also, it’s not hard to show all $1$-dimensional ones must be of the form $r=s=t=\lambda \mathrm{Id}$. In general, you can always scale a conj-rep, so I mean up to scalars.
 A: We can rephrase the question: let $G$ be the group with presentation
$$ G = \langle r, s, t \mid rs=tr, \,\,st=rs, \,\,tr=st \rangle, $$
then we are looking for finite-dimensional complex representations of $G$. Your example is interesting, essentially showing $r, s, t$ acting by the transpositions $(12), (23), (13)$ in the symmetric group $S_3$. We can rewrite the presentation for $G$ by removing the generator $t$ and replacing any occurrence of $t$ in the relations by $rsr^{-1}$. After doing this, we get the presentation
$$ G = \langle r, s \mid srs = rsr \rangle, $$
which is the Braid group $B_3$ on three strands. So really you are looking for finite-dimensional complex representations of $B_3$.
If we force both $r$ and $s$ (and hence $t$) to be involutions, then $G$ becomes the symmetric group $S_3$, and as you have pointed out this group has three irreducible representations: the trivial representation, the sign representation, and the two-dimensional representation sitting inside the three-dimensional permutation representation.
On the other hand, $B_3$ is an infinite group and so may (and indeed does) have infinitely many isomorphism classes of irreducible representations. It is a difficult problem to classify the irreducible representations of $B_n$, but maybe this has been solved for $n = 3$.
