# Prove that $\det(AB-BA)=0$

Let $$A,B$$ be two $$3 \times 3$$ matrices with complex entries such that $$(A-B)^2=O_3$$ Prove that $$\det(AB-BA)=0$$

I tried to prove this with ranks. I denoted $$X=A-B$$ and thus $$X^2=O_3$$ which means that $$\det X=0$$ and $$\operatorname{rank}X \leq 2$$. Then, I wrote $$AB-BA=(X+B)B-B(X+B)=XB-BX$$ and finally I used $$\operatorname{rank}(M \pm N) \leq \operatorname{rank}M+\operatorname{rank}N$$ and Frobenius's inequality in order to get $$\operatorname{rank}(XB-BX) \leq \operatorname{rank}BX+\operatorname{rank}XB \leq \operatorname{rank}X+\operatorname{rank}BXB$$ and if we knew that $$\operatorname{rank}BXB=0$$, the problem would be solved. However, I don't quite know if the latter is true.

• I reckon that $\text{rank }X\le1$. Dec 26, 2017 at 21:50
• Oh yes indeed, it follows from Sylvester's inequality: $\operatorname{rank}X^2=0 \geq 2\operatorname{rank}X-3$ and thus $\operatorname{rank}X \leq 1$. The problem is then solved since $\operatorname{rank}BXB \leq \operatorname{rank}X \leq 1.$ Dec 26, 2017 at 21:59

As pointed above by @Lord Shark the Unknown (whose comment struck me, pointing the right way) we have from Sylvester's inequality: $$0=\operatorname{rank}O_3=\operatorname{rank}(X^2) \geq \operatorname{rank}X+\operatorname{rank}X-3 \Rightarrow \operatorname{rank}X \leq 1.$$ Thus, $$\operatorname{rank}(XB-BX) \leq \operatorname{rank}(XB)+\operatorname{rank}(BX) \leq \operatorname{rank}X+\operatorname{rank}X \leq 2$$ and so $$\det(AB-BA)=0$$.

Here is a more or less direct, less creative solution. Since $(A-B)^2=0$, then either $A-B=0$ (in which case $AB-BA=0$), or its Jordan form is $$J=\begin{bmatrix} 0&1&0\\0&0&0\\0&0&0\end{bmatrix}.$$ So $A-B=SJS^{-1}$ for some $S$. Let $A'=S^{-1}AS$, $B'=S^{-1}BS$. Then $A'=B'+J$, and $$A'B'-B'A'=(B'+J)B'-B'(B'+J)=JB'-B'J.$$ Now check directly that $$JB'-B'J=\begin{bmatrix}B'_{31}&B'_{32}&B'_{33}-B'_{11}\\ 0&0&-B'_{21}\\ 0&0&-B'_{31} \end{bmatrix}.$$ Thus $\det(JB'-B'J)=0$. Finally, \begin{align} \det(AB-BA)&=\det(SA'S^{-1}SB'S^{-1}-SB'S^{-1}SA'S^{-1})\\ \ \\ &=\det(A'B'-B'A')=\det(JB'-B'J)=0. \end{align}

• Thank you! But why has $J$ the form you mentioned? I am sorry, but I barely know about Jordan forms at the moment. Could you please provide a good source where I could learn about them? Dec 27, 2017 at 9:36
• I think I see now: is it because $X^2$ would be the minimal polynomial of $A-B$ and since $0$ is its only eigenvalue, it means that the size of its largest Jordanian block is $2$? Dec 27, 2017 at 10:30
• The largest Jordan block has size 2, because if it had size 3 you would have $J^2\ne0$. And if it had size 1 you would have $J=0$. Dec 27, 2017 at 11:21