Unitary representations of non-compact Lie groups

Question

This question is somewhat of a continuation of this question that I had asked earlier -

Representations of a non-compact group are labeled by its maximal compact subgroup?

• I want to know when or is it always true that an unitary representation of a non-compact Lie group is infinite dimensional? If yes then why? If no then kindly give examples.

• Also when can one be sure that an infinite dimensional representation of some non-compact Lie group is labeled by a single representation of its maximal compact subgroup ?

Answer 1

Unitary repns of non-compact non-abelian Lie groups tend to be infinite-dimensional. There is a divergence between two extreme types: nilpotent versus reductive (or semi-simple). Nilpotent (or solvable) Lie groups don't have very interesting compact subgroups. Reductive or semi-simple ones, like SL(2,R), do have.

For reductive/semi-simple Lie groups such as SL(2,R), most of the irreducible unitary repns are infinite-dimensional. The exceptions are trivial, like... the trivial repn. This is because a finite dimensional unitary repn would amount to a continuous group hom of the non-compact group into a (compact) orthogonal group, which is difficult (apart from the case of R mentioned above).

Some irred unitaries of special interest are indexed by what irreds of the maximal compact K occur in them. This is the case with the holomorphic discrete series repns of SL(2,R), which are indexed by the "lowest" repn of the circle group SO(2) appearing. Mostly, however, knowing the repn as a repn of K is completely insufficient to understand the isomorphism class of the repn of the group itself. This is clear already for the ("even") principal series repns of $G=SL(2,R)$, each of which includes all the repns $\pmatrix{\cos t & \sin t\cr -\sin t & \cos t}\rightarrow e^{2int}$, with $n\in Z$, with multiplicity $1$, and the others (with odd index in the exponent) with multiplicity $0$.

Varadarajan's little Cambridge book "intro to harmonic analysis on semi-simple Lie groups" is a relatively friendly intro to such things. Finite-dimensional repns theory, and repn theory of finite groups, does not give good hints about the sort of things that happen with non-compact, non-abelian, especially reductive, Lie groups!

The nilpotent case includes things like the Heisenberg group(s), whose repn theory is well understood, is quite different from the reductive case, and perhaps much simpler.

As in a comment: inducing from a maximal compact produces repns that are too large to be irreducible, in the Lie group case. However, in a different direction, it is true that for p-adic reductive groups like $SL_2(Q_p)$, some (but only relatively unusual) irreducible unitaries are essentially obtained by inducing from the maximal compact $SL_2(Z_p)$.

Answer 2

In addition to the existing answer, let me give a proof of the fact that any smooth, finite-dimensional unitary representation $\rho: G \to GL_n(\mathbb C)$ of a non-compact, simple Lie group is always trivial.

We will crucially make use of the highly non-trivial fact that a finite-dimensional representation of a semisimple Lie group has closed image in $GL_n(\mathbb C)$, see for example Corollary 14.5.7 in Hilgert, J., Neeb, K.-H.: Structure and Geometry of Lie Groups. Springer, New York, (2013). This is false in general, as Stephen has pointed out in the comments below.

Thus, let $\rho: G \to GL_n(\mathbb C)$ be a unitary representation, i.e, we may assume that $\rho(G) \subseteq U(n)$. Since $G$ is simple and $\ker(\rho) \subseteq G$ is a normal subgroup, either $\ker(\rho) = \{1\}$ or $\ker(\rho) = G$. In the second case, $\rho$ is the trivial map and we are done.

We want to show that the first case $\ker(\rho) = \{1\}$ cannot hold, i.e, $\rho$ cannot be injective. For this, note first that $\rho(G) \subseteq U(n)$ is a closed subgroup by the first paragraph. This implies by Lie's Theorem that $\rho(G)$, equipped with the subspace topology of $G$, is a Lie group.
Since a group admits, up to diffeomorphism, at most one Lie group topology, this implies that that the injective map $\rho: G \to U(n)$ is a diffeomorphism onto its image, so that in particular, the inverse map $\rho^{-1}: \rho(G) \to G$ is continuous as well. However, $\rho(G) \subseteq U(n)$ is compact as a closed subspace of the compact group $U(n)$, so $G = \rho^{-1}(\rho(G))$ would be compact as well, a contradiction.

Edit: In my original answer, I was unaware of the fact that a faithful representation $\rho: G \to GL_n(\mathbb C)$ might not have closed image for general non-compact Lie groups $G$, which Stephen kindly reminded me of. Valter Moretti, over at Physics.stackexchange, provided me with the appropriate reference that the result holds whenever $G$ is semisimple.