# Subspace of square matrices where all matrices are nilpotent and the products of the matrices are also in the subspace.

Let $$A$$ and $$B$$ be $$3\times3$$ square matrices (maybe could also examine $$n\times n$$) and let $$U$$ be a subspace where $$A,B,AB\in U$$. All matrices in $$U$$ are nilpotent. I conjecture that all matrices in the subspace either commute up to a constant ($$AB=\alpha BA$$), or they are all strictly upper (or all strictly lower) matrices.

My reasoning is this: if $$A\in U$$ and $$B\in U$$ are nilpotent and commute up to a constant, or are strictly upper (strictly lower) matrices, then $$AB$$ and $$A+B$$ will also be nilpotent. Now of course you could have matrices $$A$$ and $$B$$ that do not commute etc. but are such that $$AB$$ and $$A+B$$ are nilpotent. But the formulation of the subspace would need $$ABA$$, $$AB^2$$, $$BAB$$, $$(A+B)A$$, $$AB+BA$$ and infinitely more to be nilpotent. I just can't see how any $$3 \times 3$$ matrix (even $$n \times n$$) could be in the subspace without fulfilling my conjecture. Maybe there is an obvious proof of my conjecture, or maybe I am just wrong and their exists an obvious counter example, or maybe it is a hard problem. Either way I would love to get a response, thanks!

Edit: My knowledge in pure math (such as abstract algebra) is severly limited, nevertheless if there exists a proof using tools from such areas I would still appreciate it a lot!

• "let $U$ be a subspace" do you mean a subalgebra? (i.e. closed not only by linear combinations but also by products) Commented Jan 18, 2023 at 15:47
• @AnneBauval To be honest with you, I have not studied much pure math and therefore not really any abstract algebra. But I guess my answer is yes if I understand correctly. If $A, B \in U$ then we should have $cA, cB, A+B, AB\in U$, where $c$ is some constant. Commented Jan 18, 2023 at 15:54
• Your hypothesis would look preferable if you said that each such subalgebra is similar to a triangular algebra. Commented Jan 18, 2023 at 16:11
• @kabenyuk Sorry, I don't understand what that means since I have not studied abstract algebra (I am an early university student, and looking at this from an linear algebra point of view). Maybe I am not fully equipped to tackle such a problem, if not there exists a counter example (like Anne's if it is indeed correct). Commented Jan 18, 2023 at 16:27

If $$U$$ an algebra of $$3\times3$$ nilpotent matrices over some field $$F$$, there exists an invertible matrix $$P$$ such that $$P^{-1}UP$$ is a subspace of strictly upper triangular matrices.
Here is the proof. Suppose first that $$U$$ has a member $$A$$ of rank $$2$$. By a change of basis, we may assume that $$A$$ is the nilpotent Jordan block. Now let $$B\in U$$. By assumption, $$A^2B=\pmatrix{b_{31}&b_{32}&b_{33}\\ 0&0&0\\ 0&0&0}, \ AB=\pmatrix{b_{21}&b_{22}&b_{23}\\ b_{31}&b_{32}&b_{33}\\ 0&0&0} \ \text{and}\ B=\pmatrix{b_{11}&b_{12}&b_{13}\\ b_{21}&b_{22}&b_{23}\\ b_{31}&b_{32}&b_{33}}$$ are nilpotent. That $$A^2B$$ is nilpotent implies that $$b_{31}=0$$. In turn, that $$AB$$ is nilpotent implies that $$b_{21}=b_{32}=0$$. Consequently, since $$B$$ is nilpotent, we must also have $$b_{11}=b_{22}=b_{33}=0$$. Hence $$B$$ is strictly upper triangular.
Now suppose that all nonzero members of $$U$$ are rank-one nilpotent matrices. Let $$A$$ be a nonzero member of $$U$$. By a change of basis, we may assume that $$A=e_1e_3^T$$. Since $$U$$ is a subspace of rank-one matrices, either $$U\subseteq\{e_1x^T:x\in F^3\}$$ or $$U\subseteq\{xe_3^T:x\in F^3\}$$. In the former case, for an arbitrary $$B=e_1x^T\in U$$ to be nilpotent, $$x_1$$ must be zero. Hence $$B$$ is strictly upper triangular. Similarly, in the latter case, for $$B=xe_3^T$$ to be nilpotent, $$x_3$$ must be zero. Hence $$B$$ is strictly upper triangular.
• (+1) Bigggie: it means that your conjecture is true: all your matrices commute up to a constant, because they are strictly upper triangular up to conjugation by some invertible matrix $P,$ and strictly upper triangular matrices commute up to a constant. Commented Jan 18, 2023 at 18:58
• I see! Nice, this does not generalize to $n \times n$ though, right? Because I don't think $4 \times 4$ strictly upper triangular matrices commute up to a constant. Commented Jan 18, 2023 at 19:53
• @Bigggie The result isn't true $n\ge4$. Let $\mathcal T$ be the subspace of all $4\times4$ strictly upper triangular matrices. Since it contains members that do not commute up to any constant, if $P$ is the permutation matrix for the transposition $(3,4)$, then $P\mathcal TP^{-1}$ contains matrices that are not strictly upper triangular and also matrices that do not commute up to any constant. Commented Jan 18, 2023 at 20:37