If $(A-B)^2=AB$, prove that $\det(AB-BA)=0$. 
Let $A,B\in M_{n}(\mathbb{Q})$. If $(A-B)^2=AB$, prove that $\det(AB-BA)=0$.

I considered the function $f:\mathbb{Q}\rightarrow \mathbb{Q}$, $f(x)=\det(A^2+B^2-BA-xAB)$ and I obtained that:
$$f(0)=\det(A^2+B^2-BA)=\det(2AB)=2^n\det(AB)$$
$$f(1)=\det(A^2+B^2-BA-AB)=\det((A-B)^2)=\det(AB)$$
$$f(2)=\det(A^2+B^2-BA-2AB)=\det((A-B)^2-AB)=\det(AB-AB)=0$$
I don't have any other idea. 
 A: In fact, there is a much stronger result.
Proposition. Let $a,b,c,d\in\mathbb{C}$. If $A,B\in M_n(\mathbb{C})$ satisfy  $aA^2+bB^2+cAB+dBA=0_n$, then, generically (for example, randomly choose $a,b,c,d$), $A,B$ are simultaneously triangularizable (for short, ST).
Proof. We follow the last part of the stewbasic's good post. We put $X=\alpha A+\beta B,Y=\gamma A+\delta B$; since generically $\alpha\delta\not= \beta\gamma$, it suffices to show that $X,Y$ are ST. 
We obtain (generically) $XY=kYX$ for some complex number $k$. Generically, $k\not=1$ and $k$ is not a primitive root of unity; then, by a result of Drazin, $X,Y$ are ST.
Remark 1. Of course, there are $a,b,c,d$ s.t. $A,B$ are not necessarily ST. For instance, consider $AB+BA=0_2$ with $A=\begin{pmatrix}1&0\\0&-1\end{pmatrix},\, B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$.
Remark 2. That is linked to the concept of quasi-commutative matrices. There are two non-equivalent definitions:
i) $A,B$ are quasi-commutative iff $AB-BA$ commute with $A,B$. By a result from McCoy, $A,B$ are always ST.
ii) $A,B$ are quasi-commutative iff $AB=kBA$ where $k$ is a complex number. This is the definition we are interested in.
A: The idea is to use $(-3\pm\sqrt 5)/2$.  We begin with
$$\begin{align}
(A+xB)(A+x'B) &= A^2 + x BA + x' AB + xx' B^2 \\ &= A^2 + (x' + x) AB+x(BA-AB)+xx'B^2.\end{align}
$$
Now we find numbers $x$, $x'$ such that $x'+x  = -3$, $xx'=1$. 
These numbers are the roots of the quadratic equation 
$$
\lambda^2 +3\lambda +1 = 0.
$$
Thus, we have $x = \frac{-3+\sqrt 5}2$ and $x'= \frac{-3-\sqrt 5}2$. 
With these, we have
$$\begin{align}
(A+xB)(A+x'B) &= A^2 -3AB + B^2 + x(BA-AB) \\ &= BA-AB + x(BA-AB) = (BA-AB)(1+x).\end{align}
$$
We now have that
$$
\det(A+xB)(A+x'B) =q \in \mathbb{Q},
$$
and 
$$
q=(1+x)^n \det(BA-AB).
$$
Then it follows by $A, B\in M_n(\mathbb{Q})$ and $(1+x)^n\in\mathbb{R} \backslash \mathbb{Q}$ that $\det(BA-AB)=0$.  

An example with $AB\neq BA$
For the example that @Jonas Meyer requested, here it is:
$$
A=\begin{pmatrix} 1 & 1 \\ 0 & \frac{-3+\sqrt 5}2\end{pmatrix}, \ \ B=\begin{pmatrix} \frac{3+\sqrt 5}2 & 1 \\ 0 & -1\end{pmatrix}.
$$
Then 
$$
A-B= \begin{pmatrix} \frac{-1-\sqrt 5}2 & 0 \\ 0 & \frac{-1+\sqrt 5}2\end{pmatrix},$$
$$
(A-B)^2 = \begin{pmatrix} \frac{3+\sqrt 5}2 & 0 \\ 0 & \frac{3-\sqrt 5}2\end{pmatrix} = AB.
$$
But, 
$$
BA =\begin{pmatrix} \frac{3+\sqrt 5}2 & \sqrt 5 \\ 0 & \frac{3-\sqrt 5}2\end{pmatrix} \neq AB.
$$
A: The statement is true without the assumption that $A,B$ have rational entries. As in i707107's answer, let
$$
  x=\frac{-3-\sqrt{5}}2,\,x'=\frac{-3+\sqrt{5}}2
$$
so that $x+x'=-3$, $xx'=1$. Let $X=A+xB$, $Y=A+x'B$. Then
$$\begin{eqnarray*}
  (1+x')XY-(1+x)YX
    &=&(x'-x)(A^2+xx'B^2)+(x'(1+x')-x(1+x))AB\\
    &&{}+(x(1+x')-x'(1+x))BA\\
    &=&(x'-x)\left[A^2+B^2+(1+x'+x)AB-BA\right]\\
    &=&(x'-x)\left[(A-B)^2-AB\right]\\
    &=&0.
\end{eqnarray*}$$
Thus $XY=kYX$ where
$$
  k=\frac{1+x}{1+x'}
$$
Note that $|k|>1$. Pick an eigenvector $v$ for $X$ whose eigenvalue $\lambda$ has maximal magnitude. Then
$$
  X(Yv)=kYXv=(k\lambda)(Yv).
$$
If $\lambda=0$ then $YXv=XYv=0$. If $\lambda\neq0$ then $|k\lambda|>|\lambda|$, so by assumption $k\lambda$ can't be an eigenvalue of $X$. This implies $Yv=0$, so again $XYv=0$ and $YXv=\lambda Yv=0$. In either case we have $(XY-YX)v=0$, so $XY-YX$ is singular. Finally since
$$
  XY-YX=(x'-x)(AB-BA),
$$
we conclude $AB-BA$ is singular.

To give a bit more insight, suppose we started with an arbitrary homogeneous degree 2 constraint on $A,B$:
$$
  c_2A^2+c_1AB+c_1'BA+c_0B^2=0.
$$
If we replace $A,B$ by commuting variables $a,b$, the corresponding polynomial would factor over $\mathbb C$:
$$
  c_2a^2+(c_1+c_1')ab+c_0b^2=(\alpha a+\beta b)(\gamma a+\delta b).
$$
Let $X=\alpha A+\beta B$ and $Y=\gamma A+\delta B$. If $A$ and $B$ commuted we'd have $XY=0$, but instead we get
$$
  XY=(\alpha\delta-c_1)[A,B]
$$
where $[A,B]=AB-BA$ is the Lie bracket. Note that $[X,Y]=(\alpha\delta-\beta\gamma)[A,B]$, so
$$
  (\alpha\delta-\beta\gamma)XY=(\alpha\delta-c_1)[X,Y].
$$
Unless a coefficient happens to vanish, this gives $XY=kYX$ for some $k$. When $k$ is not a root of unity this is quite a restrictive constraint (eg $XY$ must be singular).
