# What if there are two non-orthogonal invariant subspaces?

Let $$U_1,U_2\subseteq\Bbb R^n$$ be two invariant subspaces w.r.t. some group $$\Gamma\subseteq\mathrm O(\Bbb R^n)$$ of orthogonal matrices.

I wonder the following:

Question: If $$U_1$$ and $$U_2$$ are not orthogonal (i.e. there are $$u_i\in U_i$$ with $$\langle u_1,u_2\rangle \not=0$$), then every $$T\in\Gamma$$ restricted to $$U_1\oplus U_2$$ is already either $$\mathrm{Id}$$ or $$-\mathrm{Id}$$.

If this is true, what are the weakest assumptions we need? Do we need that the $$U_i$$ are irreducible? Does $$U_1\cap U_2 = \{0\}$$ sussfice, or maybe just $$U_1\not\subseteq U_2$$ and $$U_2\not\subseteq U_1$$.

Update

I received some very helpful answers and the statement as written above is definitely wrong. I still wonder whether there is a counterexample under the following restrictions:

1. $$\Gamma\subseteq\mathrm{O}(\Bbb R^n)$$ is a finite group.
2. The subspaces $$U_i$$ are irreducible invariant subspaces of $$\Gamma$$.
• Is it assumed that $U_1\cap U_2=\{0\}$? Feb 26, 2019 at 18:38
• @Berci Yes, sorry. Would this make a difference, except when $U_1\subseteq U_2$? Feb 26, 2019 at 18:39

Take the natural action of $$O_2(\mathbb R)$$ on the space of two-by-two matrices by left multiplication. This space has an invariant inner product $$\langle M,N\rangle = \operatorname{tr}(MN^T).$$ Take

$$U_1=\Big\{\begin{pmatrix}a&0\\b&0\end{pmatrix}\mid a,b\in\mathbb R\Big\}$$ $$U_2=\Big\{\begin{pmatrix}a&a\\b&b\end{pmatrix}\mid a,b\in\mathbb R\Big\}.$$

These spaces are not orthogonal - set $$a=b=1$$ in both.

• Wow, that was unexpected. Can we extract from that a counterexample for a finite group $\Gamma\subset\mathrm O(\Bbb R^n)$? Feb 26, 2019 at 19:16

The statement as it stands is certainly not true. Just pick any $$n>1$$, set $$U_1=U_2=\mathbb R^n$$ and pick any $$\Gamma$$ that contains at least one non-diagonalisable orthogonal matrix over $$\mathbb R$$.

The assertion is true, however, if every nonzero vector in $$U_1\ne0$$ is not orthogonal to every nonzero vectors in $$U_2\ne0$$. In this case, pick a nonzero vector $$u\in U_1$$. If $$U_2$$ contains two linearly independent vectors $$v,w$$, then by assumption, $$c_1=\langle u,v\rangle$$ and $$c_2=\langle u,w\rangle$$ are nonzero. But then we have $$\langle u,\ c_2v-c_1w\rangle=0$$, which is a contradiction to our non-orthogonality assumption.

Therefore $$U_2$$ must be one-dimensional. Similarly for $$U_1$$. Hence $$U_1,U_2$$ are eigenspaces. As eigenspaces for different eigenvalues of an orthogonal matrix are orthogonal to each other, $$U_1$$ and $$U_2$$ must be eigenspaces for the same eigenvalues. Now, over $$\mathbb R$$, the eigenvalues of an orthogonal matrix can only be $$1$$ or $$-1$$. Hence the restriction of each orthogonal transform in the group on $$U_1\oplus U_2$$ is $$\pm I$$.

• Thank your for the enlightening proof. Do you know of an example where the $U_i$ are irreducible invariant subspaces of some finite orthogonal matrix group? Feb 26, 2019 at 19:36
• @user1551 You are absolutely right, I must have been confused with the complex case. Mar 7, 2019 at 20:11

With the help of the provided answers, I was able to find a counterexample with a finite group acting on $$\Bbb R^3$$, and invariant subspaces with $$U_1\cap U_2=\{0\}$$.

Choose $$U_1=\mathrm{span}\{e_1,e_2\}$$ and $$U_2=\mathrm{span}\{e_1+e_3\}$$. These are obviously non-orthogonal and only intersect in $$\{0\}$$. Moreover, they are invariant subspaces of

$$T:=\begin{pmatrix} 1 & \phantom+0 & 0 \\ 0 & -1 & 0 \\ 0 & \phantom+0 & 1\end{pmatrix}\not\in\{\mathrm{Id},-\mathrm{Id}\}.$$

We can simply take the group $$\Gamma=\{\mathrm{Id}, \,T\}$$.

However, $$U_1$$ is not an irreducible invariant subspaces of $$\Gamma$$ as it contains the invariant subspaces $$\mathrm{span}\{e_1\}$$ and $$\mathrm{span}\{e_2\}$$.

Here is an example for which the $$U_i$$ are irreducible invariant subspaces. Let $$\bar \Gamma\subseteq\mathrm{O}(\Bbb R^2)$$ be some (sufficiently large) finite group of orthogonal transformations in the plane, e.g. $$\bar\Gamma=D_3$$. We consider the group

$$\Gamma:=\left\{ \begin{pmatrix} T & 0 \\ 0 & T\end{pmatrix} \;\middle\vert\; T\in\bar\Gamma\,\right\}\subseteq\mathrm{O}(\Bbb R^4).$$

This group has the following irreducible invariant subspaces

$$\mathrm{span}\{e_1,e_2\},\quad\mathrm{span}\{e_3,e_4\},\quad\mathrm{span}\{e_1+e_3,e_2+e_4\}.$$

While the first two are orthogonal, none of these is orthogonal to the last one.