# Intersection of two invariant subspaces

I need ro find all subspaces of $$\mathbb{R}^3$$ that are invariant simultaneously with respect to two linear transformations defined by matrices $$A=\begin{pmatrix} 5& -1& -1\\ -1& 5& -1\\ -1& -1& 5 \end{pmatrix}$$ and $$B=\begin{pmatrix} -6& 2& 3\\ 2& -3& 6&\\ 3& 6& 2 \end{pmatrix}$$ I found it eigenvalues and eigenvectors: for A $$\lambda_1=3, v_1=(1,1,1)^T\\ \lambda_2=\lambda_3=6, v_2=(1,-1,0)^T+(1,0,-1)^T$$ Invariant subspace: $${R}^3, \begin{Bmatrix}{0}\end{Bmatrix},<(1,1,1)^T> and <(1,-1,0)^T+(1,0,-1)^T>$$

for B: f$$\lambda_1=7, v_1=(1,2,3)^T\\ \lambda_2=\lambda_3=-7, v_2=(-2,1,0)^T+(-3,0,1)^T$$ Invariant subspace: $${R}^3, \begin{Bmatrix}{0}\end{Bmatrix}<(1,2,3)^T> and <(-2,1,0)^T,(-3,0,1)^T>$$ I know, that the answer is $$\mathbb {R}^3, \begin{Bmatrix}{0}\end{Bmatrix},<(1,-2,1)^T>, <(1,1,1)^T,(1,2,3)^T>$$ But I can't get it. I understand why there is $$\mathbb {R}^3 and \begin{Bmatrix}{0}\end{Bmatrix}$$

• Do you know Jordan normal form? Jun 12, 2020 at 14:04
• @FilippoGiovagnini Yes, I know Jun 12, 2020 at 14:11
• Every linear operator on $\mathbb{R}^3$ fixes $\{0\}$ and fixes $\mathbb{R}^3$ so these are always invariant subspaces. So your two lists of the invariant subspaces of $A$ and of $B$ are both missing these (rather trivial) ones. Jun 12, 2020 at 14:14
• @ancientmathematician You're right. I forgot to write them. But I know they should be there Jun 12, 2020 at 14:18
• Your eigenvectors for $A$ are wrong: try $(1,-1,0)$ and $(1,0,-1)$. Jun 12, 2020 at 14:19

I assume that all your calculations are now correct.

To find the invariant subspaces let us look for those of each possible dimension.

We have agreed that the dimension $$0$$ case gives $$\{0\}$$ and nothing else.

We have agreed that the dimension $$3$$ case gives $$\mathbb{R}^3$$ and nothing else.

For the dimension $$1$$ case we are looking for a common eigenvector of $$A$$ and $$B$$: clearly it is neither a multiple of $$(1,1,1)$$ nor of $$(1,2,3)$$. So if such exists it lies in the intersection of the two $$2$$-dimensional eigenspaces. That is, it will be some $$(x,y,z)$$ such that $$x-y-z=0$$ and $$x-2y-3z=0$$: that is it is a multiple of $$(1,-2,1)$$.

For the dimension $$2$$ case let $$U$$ be a $$2$$ dimensional subspace which is invariant under both $$A$$ and $$B$$. Note that the $$2$$-dimensional eigenspace of $$A$$ is not $$B$$-invariant, and that the $$2$$-dimensional eigenspace of $$B$$ is not $$A$$-invariant.

Suppose that $$U$$ does not contain $$(1,1,1)$$; then $$\mathbb{R}^3=\langle(1,1,1)\rangle\oplus U$$. Then $$A$$ restricted to $$U$$ must have characteristic polynomial dividing $$(X-3)(X-6)^2$$, and as the eigenvalue $$3$$ is already accounted for on $$\langle(1,1,1)\rangle$$ the characteristic polynomial of $$A$$ restricted to $$U$$ is $$(X-6)^2$$, and so $$U$$ must be the $$2$$-dimensional eigenspace of $$A$$: which we know is not so. So $$U$$ contains $$(1,1,1)$$.

A similar argument shows that $$U$$ contains $$(1,2,3)$$.

All that remains now is to check that the subspace $$U=\langle(1,1,1),(1,2,3)\rangle$$ actually is both $$A$$- and $$B$$-invariant. This is an easy calculation.