# When does $P\text{SO}P^{-1} \subseteq \text{SO}$?

Let $$P \in \text{GL}_n^{+}(\mathbb {R})$$. Suppose that $$P\cdot \text{SO}(n)\cdot P^{-1} \subseteq \text{SO}(n)$$.

Is it true that $$P \in \lambda \text{SO}(n)$$ for some $$\lambda \in \mathbb{R}$$?

I know that every matrix that commutes with $$\text{SO}(n)$$ must be in $$\lambda \text{SO}(n)$$ (and for $$n >2$$ it must be a multiple of the identity), but this is not the same question.

Edit:

In a previous version, I only required $$P \in \text{GL}_n(\mathbb {R})$$ instead of $$\text{GL}_n^{+}(\mathbb {R})$$. In that case any matrix in $$\text{O}(n)$$ would satisfy the requirements, so $$P$$ is not necessarily a multiple of special orthogonal matrix (in even dimensions).

I guess that a morally equivalent question would be to assume only $$P \in \text{GL}_n(\mathbb {R})$$, but to require $$P\cdot \text{O}(n)\cdot P^{-1} \subseteq \text{O}(n)$$. Then is it true that $$P \in \lambda \text{O}(n)$$ for some $$\lambda \in \mathbb{R}$$?

• No, for a stupid reason : in even dimension, $O(n)_{-}\neq -SO(n)$, and clearly $P\in O(n)$ works – Max Feb 12 at 12:29
• Thanks, I forgot to require that $P$ should have a positive determinant. I have edited the question accordingly. – Asaf Shachar Feb 12 at 12:35

Changing your notation slightly, let's assume $$A \in \operatorname{GL}_n^{+}(\mathbb{R})$$ such that $$A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{SO}_n(\mathbb{R})$$. We will prove that indeed we must have $$A = \lambda U$$ for $$\lambda > 0$$ and $$U \in \operatorname{SO}_n(\mathbb{R})$$. First, using polar decomposition we can write $$A = PU$$ where $$U \in \operatorname{SO}_n(\mathbb{R})$$ and $$P$$ is positive definite. Then $$A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} = P \cdot U \cdot \operatorname{SO}_n(\mathbb{R}) \cdot U^{-1} \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}).$$

Since $$Q \mapsto U \cdot Q \cdot U^{-1}$$ is an automorphism of $$\operatorname{SO}_n(\mathbb{R})$$, we get $$P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}).$$

Now, it is enough to show that $$P = \lambda I$$. Since $$P$$ is symmetric and positive, we can write $$P = Q^{-1} \Sigma Q$$ with $$Q \in \operatorname{SO}_n(\mathbb{R})$$ and $$\Sigma$$ diagonal with positive entries. Then

$$P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} = Q^{-1} \cdot \Sigma \cdot Q \cdot \operatorname{SO}_n(\mathbb{R}) \cdot Q^{-1} \cdot \Sigma^{-1} \cdot Q \subseteq \operatorname{SO}_n(\mathbb{R}).$$

Again, using the fact that conjugation is an automorphism, we get

$$\Sigma \cdot \operatorname{SO}_n(\mathbb{R}) \cdot \Sigma^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}).$$

Write $$\Sigma = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$$. Assuming $$n \geq 3$$, the group $$A_n$$ of even permutations acts transitively on $$\{ 1, \dots, n \}$$ so for any $$1 \leq i < j \leq n$$ you have a special orthogonal permutation matrix $$Q = Q_{i,j} \in \operatorname{SO}_n(\mathbb{R})$$ which satisfies $$Qe_i = e_j$$. Then $$(\Sigma Q \Sigma^{-1})(e_i) = \frac{\lambda_i}{\lambda_i} e_j$$ has norm one iff $$\lambda_i = \lambda_j$$ which shows that all the diagonal entries $$\lambda_i$$ must be the same so $$\Sigma = \lambda I$$ for $$\lambda > 0$$. When $$n = 2$$, you cannot use a permutation matrix but you can choose instead a non-diagonal rotation matrix

$$Q = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \in \operatorname{SO}_2(\mathbb{R})$$

with $$\sin \theta \neq 0$$ and then $$(\Sigma^{-1} Q \Sigma)(e_1) = \cos^2 \theta \cdot e_1 + \frac{\lambda_2}{\lambda_1} \sin^2 \theta \cdot e_2$$ has norm one iff $$\lambda_1 = \lambda_2$$ which leads to the same conclusion. The remaining case $$n = 1$$ can be trivially verified.

The same argument shows that if $$A \in \operatorname{GL}_n(\mathbb{R})$$ satisfies $$A \cdot \operatorname{O}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{O}_n(\mathbb{R})$$ then $$A = \lambda U$$ for $$U \in \operatorname{O}_n(\mathbb{R})$$ and $$\lambda \neq 0$$. In this case, you don't even need to handle the case $$n = 2$$ separately.

• Nice. This looks similar to the proof in the document linked from Valette’s answer to the question at mathoverflow.net/questions/83694/… but I didn’t inspect it carefully. – Ben Feb 13 at 3:26

You are asking about what is the normalizer of $$SO(n)$$ in the general linear group.

According to groupprops, the normalizer of $$O(n)$$ in $$GL_n(\mathbb R)$$ is the orthogonal similitude group of $$AA^t = \lambda I$$ for all $$\lambda$$ nonzero. Such matrices can be rescaled to be in $$O(n)$$ but not necessarily $$SO(n)$$ if the determinant is negative.

There are two proofs at Normalizers of maximal compact groups, on mathoverflow.

I tried to fill in a few details to the first proof in the case you are interested in:

• Let $$GL_n^+$$ act on the set of inner products on $$\mathbb R^n$$. $$SO(n)$$ fixes precisely the scalar multiples the standard form, this set denoted by $$S \cong \mathbb R_+^\times$$. (This is equivalent to showing the centralizer of $$SO(n)$$ is scalar matrices, which I see you already did.)

• The stabilizer of $$S$$ is the normalizer $$N$$ of $$SO$$ in $$GL^+$$, and acts on $$S$$. This is because if $$x \in SO_n$$ and $$g \in N$$ and we consider the form $$(g\_,g\_)$$ then acting on this by $$x$$ gives $$x(g\_,g\_) = (gx\_,gx\_) = (x'g\_,x'g\_) = g.(x'\_,x'\_) = g.(\_,\_) = (g\_,g\_)$$ So $$g.(\_,\_) \in S$$ if $$(\_,\_) \in S$$. We don't need the other containment $$stab(S) \subset N$$.

• Finally if $$A \in GL_n$$ stabilizes $$S$$ then $$A(\_,\_) = c(\_,\_)$$ for some $$c > 0$$, and we can rescale to $$A' \in S$$, so $$\mathbb R_+^\times SO(n) = stab(S) \supset N$$

The second proof at that answer reduces from showing the normalizer of $$SO(n)$$ in $$GL^+(n)$$ is $$\mathbb R_+^\times SO(n)$$ to showing $$SO(n)$$ is self-normalizing in $$SL(n)$$, which follows trivially from being a maximal subgroup in $$SL(n)$$ (which is proved in a linked paper).

• Thank you. Do you know another reference for this fact? The reference you gave does not contain a proof. – Asaf Shachar Feb 12 at 13:52
• @AsafShachar Added a link to an overflow question where you can find the proof. – Ben Feb 12 at 16:17

As you phrased your question, the answer is no. For $$P \in O(n)$$ we also have $$P\cdot A \cdot P^{-1} \in SO(n)$$ for all $$A \in SO(n)$$.

• Thanks, I forgot to require that $P$ should have a positive determinant, my apologies. I have edited the question accordingly. Indeed, my original (sloppy) phrasing has a (not very exciting) negative answer. – Asaf Shachar Feb 12 at 12:36