Let $P \in \text{GL}_n^{+}(\mathbb {R})$. Suppose that $P\cdot \text{SO}(n)\cdot P^{-1} \subseteq \text{SO}(n)$.

Is it true that $P \in \lambda \text{SO}(n)$ for some $\lambda \in \mathbb{R}$?

I know that every matrix that commutes with $\text{SO}(n)$ must be in $\lambda \text{SO}(n)$ (and for $n >2$ it must be a multiple of the identity), but this is not the same question.


In a previous version, I only required $P \in \text{GL}_n(\mathbb {R})$ instead of $\text{GL}_n^{+}(\mathbb {R})$. In that case any matrix in $\text{O}(n)$ would satisfy the requirements, so $P$ is not necessarily a multiple of special orthogonal matrix (in even dimensions).

I guess that a morally equivalent question would be to assume only $P \in \text{GL}_n(\mathbb {R})$, but to require $P\cdot \text{O}(n)\cdot P^{-1} \subseteq \text{O}(n)$. Then is it true that $P \in \lambda \text{O}(n)$ for some $\lambda \in \mathbb{R}$?

  • $\begingroup$ No, for a stupid reason : in even dimension, $O(n)_{-}\neq -SO(n)$, and clearly $P\in O(n)$ works $\endgroup$ – Max Feb 12 at 12:29
  • $\begingroup$ Thanks, I forgot to require that $P$ should have a positive determinant. I have edited the question accordingly. $\endgroup$ – Asaf Shachar Feb 12 at 12:35

Changing your notation slightly, let's assume $A \in \operatorname{GL}_n^{+}(\mathbb{R})$ such that $A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{SO}_n(\mathbb{R})$. We will prove that indeed we must have $A = \lambda U$ for $\lambda > 0$ and $U \in \operatorname{SO}_n(\mathbb{R})$. First, using polar decomposition we can write $A = PU$ where $U \in \operatorname{SO}_n(\mathbb{R})$ and $P$ is positive definite. Then $$ A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} = P \cdot U \cdot \operatorname{SO}_n(\mathbb{R}) \cdot U^{-1} \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Since $Q \mapsto U \cdot Q \cdot U^{-1}$ is an automorphism of $\operatorname{SO}_n(\mathbb{R})$, we get $$ P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Now, it is enough to show that $P = \lambda I$. Since $P$ is symmetric and positive, we can write $P = Q^{-1} \Sigma Q$ with $Q \in \operatorname{SO}_n(\mathbb{R})$ and $\Sigma$ diagonal with positive entries. Then

$$ P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} = Q^{-1} \cdot \Sigma \cdot Q \cdot \operatorname{SO}_n(\mathbb{R}) \cdot Q^{-1} \cdot \Sigma^{-1} \cdot Q \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Again, using the fact that conjugation is an automorphism, we get

$$ \Sigma \cdot \operatorname{SO}_n(\mathbb{R}) \cdot \Sigma^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Write $\Sigma = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$. Assuming $n \geq 3$, the group $A_n$ of even permutations acts transitively on $\{ 1, \dots, n \}$ so for any $1 \leq i < j \leq n$ you have a special orthogonal permutation matrix $Q = Q_{i,j} \in \operatorname{SO}_n(\mathbb{R})$ which satisfies $Qe_i = e_j$. Then $(\Sigma Q \Sigma^{-1})(e_i) = \frac{\lambda_i}{\lambda_i} e_j$ has norm one iff $\lambda_i = \lambda_j$ which shows that all the diagonal entries $\lambda_i$ must be the same so $\Sigma = \lambda I$ for $\lambda > 0$. When $n = 2$, you cannot use a permutation matrix but you can choose instead a non-diagonal rotation matrix

$$ Q = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \in \operatorname{SO}_2(\mathbb{R})$$

with $\sin \theta \neq 0$ and then $(\Sigma^{-1} Q \Sigma)(e_1) = \cos^2 \theta \cdot e_1 + \frac{\lambda_2}{\lambda_1} \sin^2 \theta \cdot e_2$ has norm one iff $\lambda_1 = \lambda_2$ which leads to the same conclusion. The remaining case $n = 1$ can be trivially verified.

The same argument shows that if $A \in \operatorname{GL}_n(\mathbb{R})$ satisfies $A \cdot \operatorname{O}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{O}_n(\mathbb{R})$ then $A = \lambda U$ for $U \in \operatorname{O}_n(\mathbb{R})$ and $\lambda \neq 0$. In this case, you don't even need to handle the case $n = 2$ separately.

  • $\begingroup$ Nice. This looks similar to the proof in the document linked from Valette’s answer to the question at mathoverflow.net/questions/83694/… but I didn’t inspect it carefully. $\endgroup$ – Ben Feb 13 at 3:26

You are asking about what is the normalizer of $SO(n)$ in the general linear group.

According to groupprops, the normalizer of $O(n)$ in $GL_n(\mathbb R)$ is the orthogonal similitude group of $AA^t = \lambda I$ for all $\lambda$ nonzero. Such matrices can be rescaled to be in $O(n)$ but not necessarily $SO(n)$ if the determinant is negative.


There are two proofs at Normalizers of maximal compact groups, on mathoverflow.

I tried to fill in a few details to the first proof in the case you are interested in:

  • Let $GL_n^+$ act on the set of inner products on $\mathbb R^n$. $SO(n)$ fixes precisely the scalar multiples the standard form, this set denoted by $S \cong \mathbb R_+^\times$. (This is equivalent to showing the centralizer of $SO(n)$ is scalar matrices, which I see you already did.)

  • The stabilizer of $S$ is the normalizer $N$ of $SO$ in $GL^+$, and acts on $S$. This is because if $x \in SO_n$ and $g \in N$ and we consider the form $(g\_,g\_)$ then acting on this by $x$ gives $$x(g\_,g\_) = (gx\_,gx\_) = (x'g\_,x'g\_) = g.(x'\_,x'\_) = g.(\_,\_) = (g\_,g\_)$$ So $g.(\_,\_) \in S$ if $(\_,\_) \in S$. We don't need the other containment $stab(S) \subset N$.

  • Finally if $A \in GL_n$ stabilizes $S$ then $A(\_,\_) = c(\_,\_)$ for some $c > 0$, and we can rescale to $A' \in S$, so $$\mathbb R_+^\times SO(n) = stab(S) \supset N$$

The second proof at that answer reduces from showing the normalizer of $SO(n)$ in $GL^+(n)$ is $\mathbb R_+^\times SO(n)$ to showing $SO(n)$ is self-normalizing in $SL(n)$, which follows trivially from being a maximal subgroup in $SL(n)$ (which is proved in a linked paper).

  • $\begingroup$ Thank you. Do you know another reference for this fact? The reference you gave does not contain a proof. $\endgroup$ – Asaf Shachar Feb 12 at 13:52
  • $\begingroup$ @AsafShachar Added a link to an overflow question where you can find the proof. $\endgroup$ – Ben Feb 12 at 16:17

As you phrased your question, the answer is no. For $P \in O(n)$ we also have $P\cdot A \cdot P^{-1} \in SO(n)$ for all $A \in SO(n)$.

  • $\begingroup$ Thanks, I forgot to require that $P$ should have a positive determinant, my apologies. I have edited the question accordingly. Indeed, my original (sloppy) phrasing has a (not very exciting) negative answer. $\endgroup$ – Asaf Shachar Feb 12 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.