If $A$ and $B$ are $2\times 2$ matrices such that $A^2+B^2=AB$, prove that $(AB-BA)^2=O$. 
Let $A$ and $B$ be two $2 \times 2$ matrices with real entries such that $A^2+B^2=AB$ then prove that $(AB-BA)^2=O$, where $O$ is the zero matrix.

I tried with assuming two matrices but it becomes very lengthy and difficult to calculate. Then I tried to use matrix product properties but not able to solve it.
$$
A^2+B^2=AB \quad \implies \quad (AB-BA)^2=O
$$
 A: First we can use Cayley-Hamilton for a 2x2 matrix to substitute the square
$$
(AB-BA)^2=(\operatorname{tr}(AB-BA))(AB-BA)-\det(AB-BA)I
$$
Since $\operatorname{tr}(AB)=\operatorname{tr}(BA)$, $\operatorname{tr}(AB-BA)=0$.
For our goal, it suffices to show that $\det(AB-BA)=0$.

For this there is a pretty clever construction proof here
Just to recap that proof, let $\omega=e^{i2\pi/3},\omega^3=1$ be 3rd unit root. Then we have this identity using $A^2+B^2=AB$
$$
\omega(AB-BA)=-(1+\omega^2)AB -\omega BA\\
=-(A^2+B^2) -\omega^2AB-\omega BA\\
=-(A+\omega B)(A+\omega^2B)\\
=-(A+\omega B)(A+\bar\omega B)
$$
Thus we have a formula for the determinant
$$
\det(A+\omega B)\det(A+\omega^2B)=\det(\omega(BA-AB))
$$
Since $\omega^2=\bar\omega$, $A,B$ are real, $\bar{A+\omega B}=A+\omega^2 B$.
Thus LHS is a real number.
$$
\det(A+\omega B)\det(A+\omega^2B)=|\det(A+\omega B)|^2
$$
While on RHS, when we deal with 2x2 matrices
$$
\det(\omega(BA-AB))=\omega^2\det(BA-AB)
$$
$\omega^2$ has a non-zero imaginary part, thus $\det(BA-AB)=0$.
Then we prove $(AB-BA)^2=0$ by Cayley-Hamilton.

This answer is inspired by  @user1551 and @Sangchul Lee :D
A: If $v$ is an eigenvector of $B$ with eigenvalue $λ$, then $A^2v+λ^2v=λAv$,
$$
(A-λωI)(A-λ\bar ωI)v=0,~~ ω^2-ω+1=0,
$$
so that $v$ is also an eigenvector of $A$, with eigenvalue $λω$ or $λ\bar ωI$. If $B$ has 2 (strict) eigenvectors, then $A$ and $B$ are therefore simultaneously diagonalizable, thus commute, thus already $AB-BA$ is zero.
In the other case $B$ has a repeated eigenvalue which has to be real, with a real eigenvector.
There is then a real basis change so that in the new basis $B=\pmatrix{λ&1\\0&λ}$, $λ$ is real with eigenvector $e_1$. The likewise changed matrix $A$ is still real and consequently has, by the above observation, the pair of conjugate eigenvalues $λω$, $λ\bar ω$, one of them with eigenvector $e_1$. However, that is impossible, as $Ae_1$ is real and at the same time equal to the strictly complex $λωe_1$.
