How to Show Representations of an Infinite Group are not Equivalent? I have unitary representations of an infinite group (but finitely generated) on an infinite dimensional Hilbert space. I would like to show that two representations are not equivalent. I am guessing I would need to find some invariant property of an representation and then show the two representations do not share the same invariant property. However after quite a bit of searching on the internet, I could not find any invariants that apply to representations on an infinite group. All I could find were invariants of representations of finite groups.
Are there are any invariants of representations of infinite groups? Are there any other methods of proving two representations are not equivalent? Also in general, I would like to learn more about representations of infinite groups. Could you recommend some resources I could look at?
 A: To fix notation let $u_1$ and $u_2$ be two unitary representations of the group $G$ on Hilbert spaces $H_1$ and $H_2$.
One of the most elementary reasons why $u_1$ and $u_2$ might fail to be unitarily equivalent is spectra:  if there is an element
$g$ in $G$ such that the spectrum of $u_1(g)$ differs from the spectrum of $u_2(g)$, then clearly $u_1$ and $u_2$ are
not unitarily equivalent.
This is not such a fine invariant, though, since representations if infinte groups often involve only  unitary operators
with full spectrum, but of course this is a start.
The representation theory of $G$ is equivalent to the representation theory of the Banach $^*$-algebra $\ell ^1(G)$, in the
sense that each $^*$-representation of the latter restricts to a unitary representation of $G$ and,  conversely,  every
representation $u$ of $G$ integrates to a representation $\pi $ of $\ell ^1(G)$ according to the formula
$$
  \pi (f) = \sum_{g\in G}f(g) u_g, \quad \forall f \in  \ell ^1(G).
  $$
The advantage of working with $\ell ^1(G)$ is the availability of a linear structure.  For example,  the kernel of the
integrated representation $\pi $ is a two-sided ideal in $\ell ^1(G)$,  and is thus an invariant for the representation $u$
from which $\pi $ has been integrated.
The envelopping C$^*$-algebra of $\ell ^1(G)$,  called $C^*(G)$,  is also a very useful gadget because, like $\ell ^1(G)$, its representation theory is
also equivalent to the representation theory of $G$. However   it  has several  advantages due to the fact that its
representation theory is highly developped.
To be a bit more precise,
given a unitary representation $u$ of $G$, the integrated representation $\pi $ on $\ell ^1(G)$ extends to
$C^*(G)$ due to the universal properties of the envelopping C$^*$-algebra, and if $\rho $ denotes this
extension, then again the kernel of $\rho $ is an invariant of $u$.
Given the finer structure of $C^*(G)$, the kernel of
$\rho $ carries much more information about $u$.  For instance, if two unitary representations $u_1$ and $u_2$ give rise to
representations $\rho _1$ and $\rho _2$ of $C^*(G)$ with the same kernel, then $u_1$ and $u_2$ are mutually weakly contained in
each other in the sense that, for every $\xi $ in $H_1$, the  positive type function on $G$ given by
$$
  g\in   G\mapsto \langle u _1(g)\xi ,\xi \rangle \in {\mathbb C}
  $$
can be approximated by positive type functions coming from vectors in the representation space of $u_2$, and vice-versa.
Another very useful algebraic invariant is the commutant of a unitary representation $u$,  defined to be the set $u(G)'$
formed by all
bounded, linear
operators $T$ commuting with every $u_g$.
$u(G)'$ is  a von Neumann algebra and it carries a lot of information about multiplicities of sub-representations of
$u$.  For instance, $u(G)'$ is isomorphic to ${\mathbb C}$ iff $u$ is irreducible while $u(G)'$ is abelian iff $u$ is multiplicity
free.
The von Neumann algebra generated by $u(G)$, namely the  commutant of $u(G)'$, is a further very important invariant.
Given the rich theory of classification of von Neumann algebras  in types, many invariants can be extracted from $u(G)''$.
