# Non-orthogonal invariant subspaces

Let $$\Gamma\subset\mathrm O(\Bbb R^n)$$ be a finite group of orthogonal matrices. Let $$U_1,U_2\subseteq\Bbb R^n$$ be two irreducible invariant subspaces w.r.t. $$\Gamma$$ with $$U_1\cap U_2=\{0\}$$, which are not orthogonal to each other, i.e. there are $$u_i\in U_i$$ with $$\langle u_1,u_2\rangle \not=0$$.

I was sceptic about the existence of such, but you can find examples here in a previous question of mine. Thinking a bit about such subspaces, I came to the following question:

Question: Is it true, that:

1. $$\dim U_1=\dim U_2=:d$$.
2. Every other $$d$$-dimensional subspace $$U\subset U_1\oplus U_2$$ with $$U\cap U_i=\{0\}$$ is an irreducible invariant subspace as well.
3. There are two orthogonal $$d$$-dimensional irreducible invariant subspaces $$\bar U_1,\bar U_2\subset U_1\oplus U_2$$.

Update

The second statement is not correct, but should be substituted by a different one. One version was given in the answer of Joppy. I can also think about something like this: every $$u\in U_1\oplus U_2\setminus\{0\}$$ is contained in exactly one $$d$$-dimensional irreducible invariant subspace $$U\subset U_1\oplus U_2$$.

• What do you mean by an irreducible invariant subspace? Mar 7, 2019 at 12:15
• @Servaes An invariant subspace without a (non-zero) invariant proper subspace. Mar 7, 2019 at 12:17
• Why are you interested in this question?
– user526015
Mar 7, 2019 at 13:55
• @James I was for a long time under the impression that (except for some easily handled cases, see my previous question) linear spaces decompose in a unique way into mutually disjoint irreducible invariant subspaces. This impression was shattered by the answers to my previous question. I now want to understand how much of this impression can be saved, or how strangely invariant subspaces can be arranged. Mar 7, 2019 at 13:59
• Then it makes much more sense to consider the eigenspaces of matrices. Mar 7, 2019 at 14:16

Allow me to translate this into more common representation-theoretic language. Saying that you have a finite subgroup $$\Gamma \subseteq O(\mathbb{R}^n)$$ is the same as the following data:

1. A finite group $$G$$,
2. A finite-dimensional real vector space $$V$$ equipped with a representation $$\rho: G \to \operatorname{GL}(V)$$, and
3. An inner product $$\langle -, - \rangle: V \times V \to \mathbb{R}$$ which is $$G$$-invariant, in the sense that $$\langle \rho(g) v, \rho(g) u \rangle = \langle v, u \rangle$$ for all $$g \in G$$ and $$u, v \in V$$.

Let $$\{I_\lambda \mid \lambda \in \Lambda\}$$ be a complete set of irreducible real representations of $$G$$. Simply knowing that $$V$$ is a real representation means that there is a canonical decomposition of $$V$$ into isotypic components, $$V = \bigoplus_{\lambda} V_\lambda$$, where $$\lambda$$ ranges over some indexing set for the isomorphism classes of irreducible representations of $$G$$. Here the subspace $$V_\lambda$$ is defined as the sum of all subrepresentations of $$V$$ isomorphic to $$I_\lambda$$. What is interesting is that these $$V_\lambda$$ must be orthogonal to each other.

Lemma: Suppose that $$U, W \subseteq V$$ are irreducible representations, and $$\langle U, V \rangle \neq 0$$. Then $$U \cong V$$ as real representations.

Proof: Since $$\langle U, V \rangle \neq 0$$, the map $$\phi: U \to V^*, \phi(u)(v) = \langle u, v \rangle$$ is nonzero. Furthermore, the $$G$$-invariancy of the inner product ensures that $$\phi$$ is a map of representations. Since $$V \cong V^*$$ as representations, we have found a nonzero $$G$$-equivariant map $$U \to V$$. By Schur's lemma, $$U \cong V$$.

This lemma shows that all the isotypic components $$V_\lambda$$ must be orthogonal under the $$G$$-invariant inner product. The answers to the rest of your questions basically follow from knowing that decomposition:

Answers to questions: let $$U_1, U_2$$ be irreducible subrepresentations of $$V$$, such that $$\langle U_1, U_2 \rangle \neq 0$$. Then:

1. $$\dim U_1 = \dim U_2$$, since by the above they must be isomorphic representations.
2. Every other nonzero $$G$$-invariant ($$\dim U_1$$)-dimensional subspace of $$U_1 \oplus U_2$$ must be isomorphic to $$U_1$$ as a representation, and hence irreducible. ($$U_1 \oplus U_2$$ is still inside the isotypic component).
3. The orthogonal complement of $$U_1$$ inside $$U_1 \oplus U_2$$ will be a subrepresentation which is both isomorpic and orthogonal to $$U_1$$.

Let $$T\in O(V)$$ be an orthogonal matrix where $$\dim V>0$$. Write its characteristic polynomial $$P_T$$ as $$P_T=\det(XI-T)=\prod_{i=1}^k P_i^{m_i},$$ where the $$P_i$$ are distinct irreducible factors and the $$m_i>0$$ their multiplicities. Because $$P_T$$ is orthogonal it is diagonalizable, hence its minimal polynomial is $$\prod_{i=1}^kP_i$$.

For each $$i$$ define $$U_i:=\ker P_i$$ and let $$T_i$$ denote the restriction of $$T$$ to $$U_i$$. Then for each $$i$$ the minimal polynomial of $$T_i$$ is precisely $$P_i$$.

Proposition: The $$U_i$$ are pairwise orthogonal $$T$$-invariant subspaces and $$V=\bigoplus_{i=1}^k U_i$$.

For every $$u\in V$$ let $$U_u$$ denote the subspace generated by the set $$\{T^k(u):\ k\geq0\}$$, where $$T^0:=I$$. Then $$U_u$$ is the smallest $$T$$-invariant subspace containing $$u$$. It is clear that

1. If $$U$$ is a $$T$$-invariant subspace and $$u\in U$$, then $$U_u\subset U$$.
2. If $$U$$ is an irreducible $$T$$-invariant subspace and $$u\in U$$ is non-zero, then $$U_u=U$$.

Because the $$P_i$$ are pairwise coprime, if a $$T$$-invariant subspace $$U$$ contains some element $$u=\sum_{i=1}^ku_i \qquad\text{ with }\ u_i\in U_i\ \text{ for each }1\leq i\leq k,$$ then it also contains $$u_i$$ for each $$1\leq i\leq k$$, and hence it contains the $$T$$-invariant subspace $$U_{u_i}\subset U_i$$. It follows that every irreducible $$T$$-invariant subspace is a subspace of some $$U_i$$. Because the $$U_i$$ are pairwise orthogonal it follows that non-orthogonal irreducible $$T$$-invariant subspaces are subspaces of the same $$U_i$$ for some $$i$$.

So let $$U_1$$ and $$U_2$$ be two non-orthogonal irreducible $$T$$-invariant subspaces of $$U$$ with $$U_1\cap U_2=0$$. Then without loss of generality the minimal polynomial of $$T$$ is an irreducible polynomial $$P$$.

For $$u\in U$$ let $$T_u$$ denote the restriction of $$T$$ to $$U_u$$. Then $$P(T_u)=0$$ so the minimal polynomial of $$T_u$$ divides $$P$$. But $$P$$ is irreducible, so the minimal polynomial of $$T_u$$ is also $$P$$ (unless $$u=0$$, then it is $$1$$). This implies that $$\dim U_u=\deg P$$, and hence every non-zero irreducible $$T$$-invariant subspace has dimension $$\deg P$$. In particular $$\dim U_1=\dim U_2=\deg P$$, proving the first statement.

The second statement holds if $$d=1$$ but fails if $$d>1$$:

If $$d=1$$ then for every $$1$$-dimensional subspace $$U\subset U_1\oplus U_2$$ we have $$U=\langle u\rangle=U_u$$ for every non-zero $$u\in U$$. This shows that every $$1$$-dimensional subspace of $$U_1\oplus U_2$$ is $$T$$-invariant, and of course it is irreducible.

For $$d=2$$ this fails; a counterexample for $$n=4$$ is given by the matrix $$T:=\begin{pmatrix} 0&-1&0&0\\ 1&\hphantom{-}0&0&0\\ 0&0&0&-1\\ 0&0&1&\hphantom{-}0 \end{pmatrix},$$ with the non-orthogonal irreducible $$T$$-invariant subspaces $$U_1:=\langle e_1,e_2\rangle \qquad\text{ and }\qquad U_2:=\langle e_1+e_3,e_2+e_4\rangle.$$ Here, the $$2$$-dimensional subspace $$\langle e_3,e_4\rangle\subset U_1\oplus U_2$$ is not $$T$$-invariant.

What is true, is that for every $$u\in U_1\oplus U_2$$ the subspace $$U_u$$ is irreducible and $$T$$-invariant, and moreover that every (non-zero) irreducible $$T$$-invariant subspace of $$U_1\oplus U_2$$ is of this form.

Note that $$d>2$$ does not occur over the real numbers, because there are no irreducible polynomials $$P\in\Bbb{R}[X]$$ with $$\deg P>2$$.

1. There are two orthogonal $$d$$-dimensional irreducible invariant subspaces $$\bar U_1,\bar U_2\subset U_1\oplus U_2$$.

This is true, and even stronger; there exists a $$d$$-dimensional $$T$$-invariant subspace $$U_2'\subset U_1\oplus U_2$$ that is orthogonal to $$U_1$$. For $$d=1$$ this is easier to see:

If $$d=1$$ then for $$i\in\{1,2\}$$ let $$u_i\in U_i$$ with $$||u_i||=1$$. Then $$U_i=\langle u_i\rangle$$, and setting $$u:=u_1-\frac{1}{\langle u_1,u_2\rangle}u_2 \qquad\text{ yields }\qquad \langle u_1,u\rangle=0,$$ so $$U_u=\langle u\rangle$$ is such a subspace.

For $$d=2$$ I think we can imitate the construction for $$d=1$$ with some adjustments, but I haven't got a proof (yet).

• Thanks for your very elaborating answer. I suspect that parts can be used to prove the same for whole matrix groups. I think that the general case of matrix groups is slightly more complicated as the irreducible invariant subspaces can have any dimension. One small remark to your answer to 2.: the intersection of $\langle e_1,e_2\rangle$ and $\langle e_1,e_3\rangle$ is not $\{0\}$. But I got the general idea, and it is clear how to make it work. Mar 7, 2019 at 17:37
• My pleasure, it was a nice exercise to straighten out my own thoughts, so +1 to you. I have corrected the mistake in my answer to 2; the subspace $\langle e_3,e_4\rangle$ does the trick. For whole matrix groups the situation only becomes simpler; the invariant subspaces only become smaller, and the answer to your questions remain exactly the same. Mar 7, 2019 at 18:00
• The last part of your comment is unfortunately not true. More matrices make the invariant subspaces larger, as they "mix" vectors from the formerly invariant subspaces. E.g. consider the group $\Gamma$ of all permutation matrices on $\Bbb R^n$. It has only two invariant subspaces: $\langle (1,...,1)\rangle$ of dimension one, and its orthogonal complement of dimension $n-1$. Mar 7, 2019 at 18:05
• I see what you mean now by invariant subspaces w.r.t. a subgroup. Then yes, this quickly becomes horrible as $n$ increases; there is little hope of saying anything sensible unless you have particular small $n$ in mind, or a particular subgroup $\Gamma$. Mar 7, 2019 at 18:17

Here is an elementary proof of $$\dim U_1=\dim U_2$$:

Proof.

Denote by $$\pi_i$$ the ortho-projector onto $$U_i$$. It is not hard to see that $$\pi_i$$ commutes with all $$T\in\Gamma$$ (e.g. by looking at their eigenspaces). Then, $$\pi_2 (U_1)$$ is $$\Gamma$$-invariant:

$$T(\pi_2 (U_1))= \pi_2(TU_1)=\pi_2(U_1),\quad\text{for all T\in\Gamma}.$$

Now, $$\pi_2(U_1)\subseteq U_2$$ is a $$\Gamma$$-invariant subspace of $$U_2$$. Since the $$U_i$$ are non-orthogonal, $$\pi_2(U_1)\not=0$$. But since $$U_2$$ is irreducible, we must have $$\pi_2(U_1)=U_2$$ already.

Equivalently, one sees that $$\pi_1(U_2)=U_1$$. Since projections can at most decrease the dimension, we have $$\dim(U_1)=\dim(U_2)$$.

$$\square$$