The action of $SL(n,\Bbb{R})$ on the tangent space of $SL(n,\Bbb{R})/SO(n)$.

Question

SETUP. It is a standard result that $\text{GL}(n,\Bbb{R})/O(n)$ is isomorphic to the set $P'$ of positive definite $n\times n$ matrices, as manifolds: the basic idea is that $\text{GL}(n,\Bbb{R})$ acts on $P'$ by $g.p = gpg^T$, the RHS interpreted as multiplication of matrices. It is an easy check that this is well defined, and it is transitive by standard linear algebra arguments. Furthermore, the stabilizer of the point $I\in P$ is clearly $O(n)$.

The same argument holds verbatim for $\text{SL}(n,\Bbb{R})/SO(n)$ and $P$, the set of positive definite matrices of determinant $1$, with the same action. Let $G=\text{GL}(n,\Bbb{R})$ or $\text{SL}(n,\Bbb{R})$, $K=O(n)$ or $SO(n)$ and $P$ the pos. def. $n\times n$ matrices, resp. those of determinant $1$.

The action of $G$ on $P$ gives us a map $G\to\text{Diff}(P)$ given by $g\mapsto \psi_g$, where $\psi_g(p) = gpg^T$. Taking the differential of $\psi_g$ and fixing the base point $I$, we get a map of tangent spaces $d\psi_g:T_I P \to T_{gg^T}P$. It is a standard result that for the Cartan decomposition $\frak{g}=k\oplus p$, with $\frak{k}$ the Lie algebra of $K$ and $\frak{p}$ the (edit: vector space of matrices with $\text{exp}(\mathfrak{p})=P$), we have a natural isomorphism $\mathfrak{p}\xrightarrow{\sim}T_I P$, and since all tangent spaces of $P$ are isomorphic, $d\psi_g$ should be able to be interpreted as a Lie algebra endomorphism (in fact an automorphism) of $\mathfrak{p}$.

QUESTION. In short, the action of $G$ on $P$ should descend to an action on $\frak{p}$. The story above is quite general, but I would like to know explicitly what this action is in the case of $G=\text{SL}(n,\Bbb{R})$ and the particular action described above.

MY ATTEMPT. For $G=\text{GL}(n,\Bbb{R})$, I believe I found the answer. Firstly, $P$ is an open submanifold of $\frak{p}$, since (positive definite) $\Leftrightarrow $ (symmetric, $\det>0$, $\text{trace}>0$), and this is an open condition on the set of symmetric matrices $\frak{p}$. Also, $\mathfrak{p}\cong \Bbb{R}^k$ in a natural way, and so it is easy to interpret $\frak{p}$ as the tangent space of $P$. Indeed, as derivations, $X\in \frak{p}$ maps to $D_X|_I$, which is given by $$D_X|_I f = \frac{d}{dt}|_{t=0}f(I+tX).$$ In this formulation, it should also be clear that the isomorphism of the tangent spaces $T_IP\cong T_p P$ is just the identity on $\frak{p}$. Now by definition, $d\psi_g$ is given by $d\psi_g(\nu)f = \nu(f\circ \psi_g)$, so applying to the above, we see that for any $f$, $$d\psi_g(D_X|_I)f = \frac{d}{dt}|_{t=0}f\circ\psi_g(I+tX) = \frac{d}{dt}|_{t=0}f(gg^T+tgXg^T) = D_{gXg^T}|_{gg^T},$$ and so the action of $\text{GL}(n,\Bbb{R})$ descends to $\frak{p}$ in the neatest way possible, $g.X=gXg^T$. One can also check that this is in fact a well defined group action.

THE DIFFICULTY. For $\text{SL}(n,\Bbb{R})$, the above argument fails. Here $\frak{p}$ is the set of symmetric matrices with trace $0$, and so $P$ is not naïvely included in this set. It might still be an open submanifold, but I haven't proven it. Most egregious is the fact that the action above is simply not well defined: you will find plenty of examples of $gXg^T$ with $\text{tr} X=0$ but $\text{tr} (gXg^T)\neq 0$.

IDEAS. Can you modify my proof to work, or can you maybe explicate an embedding $P$ into $\frak{p}$ as to make the action of $\text{SL}(n,\Bbb{R})$ explicit?

Another way may be to bypass the use of $P$ entirely and look at the action of $G$ on $G/K$ by left multiplication, and explicating its differential as an automorphism of $\frak{p}$.

Answer 1

Since you asked about the case $n=2$: If such a representation exists, it is a nontrivial linear (real) 2-dimensional representation $\rho$. Moreover, since the center of $SL(2,\mathbb R)$ acts trivially on $P$, it has to be a representation factoring through a representation of $PSL(2, \mathbb R)$ (the quotient of $SL(2,\mathbb R)$ by its order 2 central subgroup). But the lowest-dimensional nontrivial representation of $PSL(2, \mathbb R)$ is the adjoint representation, which is 3-dimensional. Thus, a representation $\rho$ does not exist. In this argument I used one nontrivial fact, the classification of irreducible finite-dimensional representations of $SL(2)$ by their highest weights. This would be discussed in any textbook on representation theory of Lie groups or/and Lie algebras, for instance, "Representation Theory: A First Course" by Fulton.

Answer 2

You are making certain (very interesting !) mistakes. This is not quite my field, so I apologize if I say something imprecise.

The overall issue is the question of "the tangent space of $P$ at $p$". First of all, notice that the conditions "positive trace and determinant" only have the effect of shrinking $P$ down to an open subset, and since tangent spaces aren't affected by restricting to opens, we might as well assume that $P$ is just the symmetric matrices $S_n(\Bbb R)$ ($S_n(\Bbb R) \cap \operatorname{SL}_n(\Bbb R)$ in the second case).

The first question to be asked is this: is $P$ even a manifold ? In the case $S_n(\Bbb R)$, yes, as it is a vector space, and the tangent space anywhere is canonically identified with $P$ itself. But in the case $S_n(\Bbb R) \cap \operatorname{SL}_n(\Bbb R)$, things become more complicated (correct me if I am wrong in the following). Two submanifolds intersect as a manifold around points where the intersection is transverse. Luckily, in our case, it works.

The second point to be adressed is this: in the second case, $S_n(\Bbb R) \cap \operatorname{SL}_n(\Bbb R)$ is not closed under addition nor multiplication, so there is no obvious way to give it a Lie group structure. Thus, it does not make sense to talk about its Lie algebra. Things worked out in the first case because $S_n(\Bbb R)$ is a Lie group for addition. That being said, in both cases, automorphisms of $P$ still lead to maps between tangent spaces ; the remaining challenge is to relate those tangent spaces in order to fall back to an automorphism of $T_IP$.

I'll expand on the comment of Moishe Kohan. Given a Lie group $G$ and $g \in G$, the Lie group structure gives us a canonical way to relate $T_eG$ with $T_gG$: the map derived from multiplication by $g$. In the first case $P=S_n(\Bbb R)$, "multiplication by $M$" is just translation by $M$, for which the differential is the identity (which is why your reasoning worked in the first case). Your "canonical map" was indeed the one coming from the Lie group action. In the second case however, there is no Lie group action. You might ask "well why don't we use $\psi_g$ to map $I$ to $gg^T$ ?", which works, but has a major flaw: there might be multiple $g$'s which give the same value for $gg^T$, but have an overall different action on $P$ as a whole, and hence, even though they all map $T_I P$ to $T_{gg^T} P$, they might do so in different ways, and there is no canonical choice. Hence the "action" of $G$ on $T_IP$ depends on lots of choices which might not form a coherent system overall (in other words, you won't get an action, just a map $G \rightarrow \operatorname{Aut}(T_I P)$ as sets).