Product of nilpotent matrices.

Question

Let $A$ and $B$ be $n \times n$ complex matrices and

let $[A,B] = AB - BA$.

Question: If $A , B$ and $[A,B]$ are all nilpotent matrices,

is it necessarily true that $\operatorname{trace}(AB) = 0$?

If,in fact, $[A,B] = 0$, then we can take $A$ and $B$ to be strictly upper triangular matrices so that the answer would be yes in this very special case.

Answer 1

The answer is no.

Take $A=\left(\begin{array}{cccc} 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \end{array}\right)$, $B=XAX^{-1} = \left(\begin{array}{cccc} -1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ -1 & 1 & -1 & 1\end{array}\right)$, where we chose $X=\left(\begin{array}{cccc} 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 1 & 1 \end{array}\right)$.

Then $[A,B]= \left(\begin{array}{cccc} 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ -1 & 1 & -1 & 1 \\ 0 & 1 & -1 & 1 \end{array}\right)$ is nilpotent, but we have $AB=\left(\begin{array}{cccc} 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ -1 & 1 & -1 & 1 \\ 0 & 0 & 0 & 0 \end{array}\right)$ and $\operatorname{trace}(AB)=-1\not=0$.

Answer 2

No. While the conjecture is true for $n=2$ and computer experiments suggest that it may also be true for $n=3$, counterexamples are abundant when $n=4$. Let $A$ be the $4\times4$ Jordan block. The following $B$s are some computer generated random counterexamples that satisfy the condition $B^4=(AB-BA)^4=0$ but $\mathrm{trace}(AB)\neq0$. \begin{align*} \pmatrix{0&0&0&0\\ 0&1&1&0\\ -1&-1&-1&0\\ 1&0&0&0}, &\pmatrix{ 4& -4& 4& -6\\ 2& -2& 2& -3\\ 14&-14& 14&-13\\ 16&-16& 16&-16 },\\ \\ \pmatrix{ 1& 1& 1& 2\\ 0& 0& 0& 0\\ 1& 2& 1& 2\\ -1& -2& -1& -2\\ }, &\pmatrix{ -1& -1& 0& 0\\ 1& 1& 0& 0\\ 0& 0& 0& 1\\ 1& 1& 0& 0\\ }. \end{align*}