Non-orthogonal invariant subspaces Let $\Gamma\subset\mathrm O(\Bbb R^n)$ be a finite group of orthogonal matrices. Let $U_1,U_2\subseteq\Bbb R^n$ be two irreducible invariant subspaces w.r.t. $\Gamma$ with $U_1\cap U_2=\{0\}$, which are not orthogonal to each other, i.e. there are $u_i\in U_i$ with $\langle u_1,u_2\rangle \not=0$.
I was sceptic about the existence of such, but you can find examples here in a previous question of mine. Thinking a bit about such subspaces, I came to the following question:

Question: Is it true, that:

*

*$\dim U_1=\dim U_2=:d$.

*Every other $d$-dimensional subspace $U\subset U_1\oplus U_2$ with $U\cap U_i=\{0\}$ is an irreducible invariant subspace as well.

*There are two orthogonal $d$-dimensional irreducible invariant subspaces $\bar U_1,\bar U_2\subset U_1\oplus U_2$.



Update
The second statement is not correct, but should be substituted by a different one. One version was given in the answer of Joppy. I can also think about something like this: every $u\in U_1\oplus U_2\setminus\{0\}$ is contained in exactly one $d$-dimensional irreducible invariant subspace $U\subset U_1\oplus U_2$.
 A: Allow me to translate this into more common representation-theoretic language. Saying that you have a finite subgroup $\Gamma \subseteq O(\mathbb{R}^n)$  is the same as the following data:


*

*A finite group $G$,

*A finite-dimensional real vector space $V$ equipped with a representation $\rho: G \to \operatorname{GL}(V)$, and

*An inner product $\langle -, - \rangle: V \times V \to \mathbb{R}$ which is $G$-invariant, in the sense that $\langle \rho(g) v, \rho(g) u \rangle = \langle v, u \rangle$ for all $g \in G$ and $u, v \in V$.


Let $\{I_\lambda \mid \lambda \in \Lambda\}$ be a complete set of irreducible real representations of $G$. Simply knowing that $V$ is a real representation means that there is a canonical decomposition of $V$ into isotypic components, $V = \bigoplus_{\lambda} V_\lambda$, where $\lambda$ ranges over some indexing set for the isomorphism classes of irreducible representations of $G$. Here the subspace $V_\lambda$ is defined as the sum of all subrepresentations of $V$ isomorphic to $I_\lambda$. What is interesting is that these $V_\lambda$ must be orthogonal to each other.
Lemma: Suppose that $U, W \subseteq V$ are irreducible representations, and $\langle U, V \rangle \neq 0$. Then $U \cong V$ as real representations.
Proof: Since $\langle U, V \rangle \neq 0$, the map $\phi: U \to V^*, \phi(u)(v) = \langle u, v \rangle$ is nonzero. Furthermore, the $G$-invariancy of the inner product ensures that $\phi$ is a map of representations. Since $V \cong V^*$ as representations, we have found a nonzero $G$-equivariant map $U \to V$. By Schur's lemma, $U \cong V$.
This lemma shows that all the isotypic components $V_\lambda$ must be orthogonal under the $G$-invariant inner product. The answers to the rest of your questions basically follow from knowing that decomposition:

Answers to questions: let $U_1, U_2$ be irreducible subrepresentations of $V$, such that $\langle U_1, U_2 \rangle \neq 0$. Then:
  
  
*
  
*$\dim U_1 = \dim U_2$, since by the above they must be isomorphic representations.
  
*Every other nonzero $G$-invariant ($\dim U_1$)-dimensional subspace of $U_1 \oplus U_2$ must be isomorphic to $U_1$ as a representation, and hence irreducible. ($U_1 \oplus U_2$ is still inside the isotypic component).
  
*The orthogonal complement of $U_1$ inside $U_1 \oplus U_2$ will be a subrepresentation which is both isomorpic and orthogonal to $U_1$.
  

A: Let $T\in O(V)$ be an orthogonal matrix where $\dim V>0$. Write its characteristic polynomial $P_T$ as
$$P_T=\det(XI-T)=\prod_{i=1}^k P_i^{m_i},$$
where the $P_i$ are distinct irreducible factors and the $m_i>0$ their multiplicities. Because $P_T$ is orthogonal it is diagonalizable, hence its minimal polynomial is $\prod_{i=1}^kP_i$.
For each $i$ define $U_i:=\ker P_i$ and let $T_i$ denote the restriction of $T$ to $U_i$. Then for each $i$ the minimal polynomial of $T_i$ is precisely $P_i$.
Proposition: The $U_i$ are pairwise orthogonal $T$-invariant subspaces and $V=\bigoplus_{i=1}^k U_i$.
Proof. See proposition 4.7 of the linked reader.
For every $u\in V$ let $U_u$ denote the subspace generated by the set $\{T^k(u):\ k\geq0\}$, where $T^0:=I$. Then $U_u$ is the smallest $T$-invariant subspace containing $u$. It is clear that


*

*If $U$ is a $T$-invariant subspace and $u\in U$, then $U_u\subset U$.

*If $U$ is an irreducible $T$-invariant subspace and $u\in U$ is non-zero, then $U_u=U$.


Because the $P_i$ are pairwise coprime, if a $T$-invariant subspace $U$ contains some element 
$$u=\sum_{i=1}^ku_i
\qquad\text{ with }\ u_i\in U_i\ \text{ for each }1\leq i\leq k,$$
then it also contains $u_i$ for each $1\leq i\leq k$, and hence it contains the $T$-invariant subspace $U_{u_i}\subset U_i$. It follows that every irreducible $T$-invariant subspace is a subspace of some $U_i$. Because the $U_i$ are pairwise orthogonal it follows that non-orthogonal irreducible $T$-invariant subspaces are subspaces of the same $U_i$ for some $i$. 
So let $U_1$ and $U_2$ be two non-orthogonal irreducible $T$-invariant subspaces of $U$ with $U_1\cap U_2=0$. Then without loss of generality the minimal polynomial of $T$ is an irreducible polynomial $P$.
For $u\in U$ let $T_u$ denote the restriction of $T$ to $U_u$. Then $P(T_u)=0$ so the minimal polynomial of $T_u$ divides $P$. But $P$ is irreducible, so the minimal polynomial of $T_u$ is also $P$ (unless $u=0$, then it is $1$). This implies that $\dim U_u=\deg P$, and hence every non-zero irreducible $T$-invariant subspace has dimension $\deg P$. In particular $\dim U_1=\dim U_2=\deg P$, proving the first statement.
The second statement holds if $d=1$ but fails if $d>1$:
If $d=1$ then for every $1$-dimensional subspace $U\subset U_1\oplus U_2$ we have $U=\langle u\rangle=U_u$ for every non-zero $u\in U$. This shows that every $1$-dimensional subspace of $U_1\oplus U_2$ is $T$-invariant, and of course it is irreducible.
For $d=2$ this fails; a counterexample for $n=4$ is given by the matrix
$$T:=\begin{pmatrix}
0&-1&0&0\\
1&\hphantom{-}0&0&0\\
0&0&0&-1\\
0&0&1&\hphantom{-}0
\end{pmatrix},$$
with the non-orthogonal irreducible $T$-invariant subspaces
$$U_1:=\langle e_1,e_2\rangle
\qquad\text{ and }\qquad
U_2:=\langle e_1+e_3,e_2+e_4\rangle.$$
Here, the $2$-dimensional subspace $\langle e_3,e_4\rangle\subset U_1\oplus U_2$ is not $T$-invariant.
What is true, is that for every $u\in U_1\oplus U_2$ the subspace $U_u$ is irreducible and $T$-invariant, and moreover that every (non-zero) irreducible $T$-invariant subspace of $U_1\oplus U_2$ is of this form.
Note that $d>2$ does not occur over the real numbers, because there are no irreducible polynomials $P\in\Bbb{R}[X]$ with $\deg P>2$.


  
*There are two orthogonal $d$-dimensional irreducible invariant subspaces $\bar U_1,\bar U_2\subset U_1\oplus U_2$.
  

This is true, and even stronger; there exists a $d$-dimensional $T$-invariant subspace $U_2'\subset U_1\oplus U_2$ that is orthogonal to $U_1$. For $d=1$ this is easier to see:
If $d=1$ then for $i\in\{1,2\}$ let $u_i\in U_i$ with $||u_i||=1$.  Then $U_i=\langle u_i\rangle$, and setting
$$u:=u_1-\frac{1}{\langle u_1,u_2\rangle}u_2
\qquad\text{ yields }\qquad
\langle u_1,u\rangle=0,$$
so $U_u=\langle u\rangle$ is such a subspace.
For $d=2$ I think we can imitate the construction for $d=1$ with some adjustments, but I haven't got a proof (yet).
A: Here is an elementary proof of $\dim U_1=\dim U_2$:
Proof.
Denote by $\pi_i$ the ortho-projector onto $U_i$. It is not hard to see that $\pi_i$ commutes with all $T\in\Gamma$ (e.g. by looking at their eigenspaces). Then, $\pi_2 (U_1)$ is $\Gamma$-invariant:
$$T(\pi_2 (U_1))= \pi_2(TU_1)=\pi_2(U_1),\quad\text{for all $T\in\Gamma$}.$$
Now, $\pi_2(U_1)\subseteq U_2$ is a $\Gamma$-invariant subspace of $U_2$. Since the $U_i$ are non-orthogonal, $\pi_2(U_1)\not=0$. But since $U_2$ is irreducible, we must have $\pi_2(U_1)=U_2$ already.
Equivalently, one sees that $\pi_1(U_2)=U_1$. Since projections can at most decrease the dimension, we have $\dim(U_1)=\dim(U_2)$.
$\square$
