Let $A,B\in M_2(\mathbb{C})$ such that $A^2+B^2=3AB$. Prove or disprove that $ \det(AB+BA)=\det(2AB). $ Let $A,B\in M_2(\mathbb{C})$ such that $A^2+B^2=3AB$. Prove or disprove that
$$
\det(AB+BA)=\det(2AB).
$$
I try to start with $\det(2AB)$ and write
$$
 \det(2AB)=4\det(AB)=\frac{4}{9}\det(3AB)=\frac{4}{9}\det(A^2+B^2).
$$ 
After that it seems to me that something is missing. How can I continue from here? Thank you.
 A: The result follows immediately if $AB=BA$. So, let us assume that $AB\ne BA$. By Cayley-Hamilton theorem, $A^2$ is a linear combination of $A$ and $I$ and $B^2$ is a linear combination of $B$ and $I$. Therefore the condition $A^2+B^2=3AB$, or equivalently $AB-\frac13(A^2+B^2)=0$ can be rewritten in the form of
$$
(A+pI)(B+qI)=rI.\tag{1}
$$
By assumption, $AB\ne BA$. Hence the RHS of $(1)$ must be zero. That is, $XY=0$ when $X=A+pI$ and $Y=B+qI$. Hence one of $X$ or $Y$ is singular but nonzero (it cannot be zero because $AB\ne BA$ implies that $A$ and $B$ are not scalar matrices). Suppose $X\ne0$ is singular (the other case is similar). By a change of basis, we may assume that $X$ is already in Jordan form. However, as
\begin{aligned}
\pmatrix{0&1\\ 0&0}Y=0\quad\Rightarrow\quad Y=\pmatrix{\ast&\ast\\ 0&0},\\
\underbrace{\pmatrix{\lambda&0\\ 0&0}}_{\lambda\ne0}Y=0\quad\Rightarrow\quad Y=\pmatrix{0&0\\ \ast&\ast},
\end{aligned}
$X$ and $Y$ are simultaneously triangularised. Hence $A$ and $B$ are simultaneously triangulable and the result follows.

Remark. Using the technique applied in loup blanc's answer to a similar question, we can prove that the statement here is actually true when $A,B$ are $n\times n$. Let $w=\frac{3+\sqrt{5}}{2}$ and $w'=\frac{3-\sqrt{5}}{2}$. Then we may write $A=wX+w'Y$ and $B=X+Y$ for some $X$ and $Y$. The condition $A^2+B^2=3AB$ thus reduces to
$$
XY=kYX,\tag{2}
$$
where $0<k=\frac{w'}{w}<1$. Therefore at least one of $X$ or $Y$ is singular. If $X$ is singular (the other case is similar), $(2)$ implies that $\ker(X)$ is an invariant subspace of $Y$. Therefore $X$ and $Y$ share an eigenvector in $\ker(X)$. Continue the argument recursively, we see that $X$ and $Y$ are simultaneously triangulable. Hence $A$ and $B$ are simultaneously triangulable too and the result follows.
