# Set of homomorphisms from discrete upper triangular group into continuous u.t. group

Let $G$ be the group $$\begin{pmatrix} 1 & a_{12} & a_{13} & a_{14}\\ 0 & 1 & a_{23} & a_{24}\\ 0 & 0 & 1 & a_{34}\\ 0 & 0 & 0 & 1 \end{pmatrix}$$ where $a_{ij}\in\mathbb R$ and $\Gamma=G\cap\mathrm{GL}_4\,\mathbb Z$. What is the set of injective homomorphisms $\Gamma\to G$?

-
Where do the $a_{ij}$ live? The reals? – JSchlather Jul 24 '12 at 17:49
Yes, sorry, added. – Earthliŋ Jul 25 '12 at 1:22
Hm, so what have you tried? You certainly get the natural inclusion and then you can look at the conjugates of that subgroup. – JSchlather Jul 25 '12 at 13:03
But will conjugates alone give all injective homomorphisms? For example, in dimension 3, the set of injective homomorphisms is the 6-dimensional space $GL_2\mathbb R\times\mathbb R^2$ with $G$ acting transitively on the $\mathbb R^2$ factor, but trivially on $GL_2\mathbb R$. Including and conjugating would thus only yield $\{\text{pt}\}\times\mathbb R^2$ and not the whole set... – Earthliŋ Jul 25 '12 at 15:07

There a neat answer to the question of describing all homomorphisms $\Gamma\to G$. Malcev indeed proved (see the book by Raghunathan) a general "superrigidity" result for general unipotent groups, implying in particular that every homomorphism $\Gamma\to G_1$ has a unique extension as a continuous homomorphism $G\to G_1$ ($G_1$ any real unipotent group). Since $G$ is a simply connected Lie group, this shows that the set of continuous homomorphisms $G\to G$ can be identitied to the set of endomorphisms of its Lie algebra, which is more explicitly computable, but is not yet a linear object.
However assuming the homomorphism injective in restriction to $\Gamma$ here simplifies a bit. Namely, the Lie algebra $\mathfrak{g}$ of $G$ has the property that its center is 1-dimensional, so any non-injective homomorphism from $G$ to any Lie group will kill the center and hence be non-injective in restriction to $\Gamma$.
This shows that the set of injective homomorphisms $\Gamma\to G$ can be identified to the group $H$ of automorphisms of the Lie algebra $\mathfrak{g}$. This is a finite dimensional Lie group with finitely many components (as the group of real points of an algebraic group), whose Lie algebra $\mathfrak{h}$ can be identified to the space of derivations of the Lie algebra $\mathfrak{g}$; the latter object being even more easily computable (because unlike the set of endomorphisms, this is a linear object).
I haven't done the computation of $\mathfrak{h}$ (do it!); just note that $H$ is at least 8-dimensional, because it contains the group $T$ of conjugations by elements of the upper triangular group (which is 10-dimensional but scalar matrices as well as $(1,4)$-matrices act trivially). It also contains the involution $j:Z\mapsto Qt(Z^{-1})Q^{-1}$, where $t$ denotes transpose and $Q$ is the antidiagonal matrix with antidiagonal entries equal to 1; the latter normalizes $T$, so the group generated by $T$ and $j$ contains $T$ with index 2.