Unitary representations of non-compact Lie groups

This question is somewhat of a continuation of this question that I had asked earlier -

Representations of a non-compact group are labeled by its maximal compact subgroup?

• I want to know when or is it always true that an unitary representation of a non-compact Lie group is infinite dimensional? If yes then why? If no then kindly give examples.

• Also when can one be sure that an infinite dimensional representation of some non-compact Lie group is labeled by a single representation of its maximal compact subgroup ?

• I'm not sure exactly what you're asking in the second bullet point, but concerning the first I can tell you that it's obviously not true: How about the two-dimensional representation of $\mathbb{R}$ given by $t \mapsto \begin{bmatrix} \cos{t} &-\sin{t} \\\ \sin{t} & \cos{t}\end{bmatrix}$ or any character representation? – t.b. Jul 4 '11 at 5:57
• Maybe the question is whether evry inf. dimensional representation of a noncompact Lie-group is induced from a maximal compact subgroup? – Mark Jul 4 '11 at 8:35

Unitary repns of non-compact non-abelian Lie groups tend to be infinite-dimensional. There is a divergence between two extreme types: nilpotent versus reductive (or semi-simple). Nilpotent (or solvable) Lie groups don't have very interesting compact subgroups. Reductive or semi-simple ones, like SL(2,R), do have.

For reductive/semi-simple Lie groups such as SL(2,R), most of the irreducible unitary repns are infinite-dimensional. The exceptions are trivial, like... the trivial repn. This is because a finite dimensional unitary repn would amount to a continuous group hom of the non-compact group into a (compact) orthogonal group, which is difficult (apart from the case of R mentioned above).

Some irred unitaries of special interest are indexed by what irreds of the maximal compact K occur in them. This is the case with the holomorphic discrete series repns of SL(2,R), which are indexed by the "lowest" repn of the circle group SO(2) appearing. Mostly, however, knowing the repn as a repn of K is completely insufficient to understand the isomorphism class of the repn of the group itself. This is clear already for the ("even") principal series repns of $G=SL(2,R)$, each of which includes all the repns $\pmatrix{\cos t & \sin t\cr -\sin t & \cos t}\rightarrow e^{2int}$, with $n\in Z$, with multiplicity $1$, and the others (with odd index in the exponent) with multiplicity $0$.

Varadarajan's little Cambridge book "intro to harmonic analysis on semi-simple Lie groups" is a relatively friendly intro to such things. Finite-dimensional repns theory, and repn theory of finite groups, does not give good hints about the sort of things that happen with non-compact, non-abelian, especially reductive, Lie groups!

The nilpotent case includes things like the Heisenberg group(s), whose repn theory is well understood, is quite different from the reductive case, and perhaps much simpler.

As in a comment: inducing from a maximal compact produces repns that are too large to be irreducible, in the Lie group case. However, in a different direction, it is true that for p-adic reductive groups like $SL_2(Q_p)$, some (but only relatively unusual) irreducible unitaries are essentially obtained by inducing from the maximal compact $SL_2(Z_p)$.

In addition to the existing answer, let me give a proof of the fact that any smooth, finite-dimensional unitary representation $\rho: G \to GL_n(\mathbb C)$ of a non-compact, simple Lie group is always trivial.

We will crucially make use of the highly non-trivial fact that a finite-dimensional representation of a semisimple Lie group has closed image in $GL_n(\mathbb C)$, see for example Corollary 14.5.7 in Hilgert, J., Neeb, K.-H.: Structure and Geometry of Lie Groups. Springer, New York, (2013). This is false in general, as Stephen has pointed out in the comments below.

Thus, let $\rho: G \to GL_n(\mathbb C)$ be a unitary representation, i.e, we may assume that $\rho(G) \subseteq U(n)$. Since $G$ is simple and $\ker(\rho) \subseteq G$ is a normal subgroup, either $\ker(\rho) = \{1\}$ or $\ker(\rho) = G$. In the second case, $\rho$ is the trivial map and we are done.

We want to show that the first case $\ker(\rho) = \{1\}$ cannot hold, i.e, $\rho$ cannot be injective. For this, note first that $\rho(G) \subseteq U(n)$ is a closed subgroup by the first paragraph. This implies by Lie's Theorem that $\rho(G)$, equipped with the subspace topology of $G$, is a Lie group.
Since a group admits, up to diffeomorphism, at most one Lie group topology, this implies that that the injective map $\rho: G \to U(n)$ is a diffeomorphism onto its image, so that in particular, the inverse map $\rho^{-1}: \rho(G) \to G$ is continuous as well. However, $\rho(G) \subseteq U(n)$ is compact as a closed subspace of the compact group $U(n)$, so $G = \rho^{-1}(\rho(G))$ would be compact as well, a contradiction.

Edit: In my original answer, I was unaware of the fact that a faithful representation $\rho: G \to GL_n(\mathbb C)$ might not have closed image for general non-compact Lie groups $G$, which Stephen kindly reminded me of. Valter Moretti, over at Physics.stackexchange, provided me with the appropriate reference that the result holds whenever $G$ is semisimple.

• I don't understand how you establish that the image $\rho(G)$ is closed. It seems you claim to deduce this from the fact that $\rho$ is injective. But of course the non-compact group $\mathbf{R}$ of all real numbers embeds in the compact torus $S^1 \times S^1$ via $x \mapsto (e^{2 \pi i x},e^{2 \pi i \alpha x})$ for any irrational $\alpha$; while this is not simple it would seem your argument from the second paragraph on applies to it as well. What's really going on? – Stephen Jul 28 '18 at 13:38
• To expand a little bit more on my last comment, the example you gave was that of an immersed, non-embedded Lie subgroup. The image of a continuous, immersive group homomorphism $i: H \to G$ is called an immersed Lie subgroup of $G$. If, additionally, this immersion is a diffeomorphism, i.e, the Liegroup topology on $i(H)$ inherited by $H$ and the subgroup topology of $i(H) \subseteq G$ conincide, then $i(H)$ is an embedded Lie group. Embedded Lie groups of $G$ are precisely the closed subgroups of $G$. – H1ghfiv3 Jul 28 '18 at 20:02
• The map I wrote down is smooth. Sard's theorem guarantees that the set of critical values is measure zero (which it is). If it's true that the image of a simple Lie group by a smooth homorphism is a closed subgroup, then there has to be a reason for it that doesn't appear to have been written down yet, I think. – Stephen Jul 28 '18 at 22:04
• I made a mistake by confusing dense image sets with sets of full measure. However, your map $f: \mathbb R \to S^1 \times S^1$ is only an immersion, but not a smooth embedding, since the image $f(\mathbb R)$ is not a smooth submanifold of $S^1 \times S^1$ (for this, one can indeed use the dense image argument). I agree that I will have to clarify why one would obtain an embedding under $\rho$, not only a smooth immersion. – H1ghfiv3 Jul 29 '18 at 7:01
• " I agree that I will have to clarify why one would obtain an embedding under ρ, not only a smooth immersion. " Yes, this would seem to be the whole point. – Stephen Jul 29 '18 at 23:02