$\operatorname{Tr}(AB)=\operatorname{Tr}(A)\operatorname{Tr}(B)$ 
Problem. Show that $\operatorname{Tr}(AB)=\operatorname{Tr}(A)\operatorname{Tr}(B)$, if $A^2+3AB+B^2=BA$ and $\det(A)=0$, where all matrices involved is real valued $2\times2$ matrices.

First I rewrote the $A^2+3AB+B^2=BA$ as 
$$ A(A+3B)=B(A-B) $$
and I took the determinant. Then I obtained that
$$ \det(B)\det(A-B)=0, $$
because $\det(A)=0$. So, $\det(B)=0$ or $\det(A-B)=0$.
Case I. Then, by Cayley-Hamiton theorem, I wrote that
$$ A^2=(\operatorname{Tr}(A))A \text{ and } B^2=(\operatorname{Tr}(B))B. $$
Multiplying the above two equations, I got $(AB)^2=(\operatorname{Tr}(A)\operatorname{Tr}(B))AB$. Another use of Cayley theorem got $(AB)^2=(\operatorname{Tr}(AB))AB$. So, $\operatorname{Tr}(A)\operatorname{Tr}(B)$ must be equal with $\operatorname{Tr}(AB)$ only if $AB=BA$. But I don't have such a hypothesis.
Case II. For the case where $\det(A-B)$ I don't know how to handle this one in order to get the answear.
 A: Here is a bit messy solution:


Lemma. Let $F$ be a field with characteristic $\neq 2$ and $A$ and $B$ be $2\times 2$ be matrices over $F$.
  
  
*
  
*If $A$ and $B$ commute, then either $A = pB + qI$ or $B = pA + qI$ for some $p, q \in F$.
  
*If $A^2 = k(AB - BA)$, then $A^2 = 0$. In particular, if $k \neq 0$, then $AB = BA$.
  

We postponed the proof of this lemma to the end. By rearranging the equality in the assumption, we obtain $(A+B)^2 = 2[(A+B)A-A(A+B)]$. So, by Lemma, $(A+B)^2 = 0$ and $AB = BA$. Now we claim the following.

Claim. Either $A = 0$ or $B = kA$ for some $k$.

Once this is true, then the rest is straightforward: If $A = 0$, there is nothing to prove. If $B = kA$ for some $k$, then
$$ \operatorname{Tr}(AB)
= k\operatorname{Tr}(A^2)
= k\operatorname{Tr}(A)^2
= \operatorname{Tr}(A)\operatorname{Tr}(B), $$
where the second step follows from the identity $\operatorname{Tr}(A^2) = \operatorname{Tr}(A)^2$, which itself is a consequence of both Cayley-Hamilton theorem and $\det(A) = 0$. $\square$
So it remains to show the claim. Assume that $A \neq 0$. Since $\det(A) = 0$, we may utilize Lemma to write
$$A+B = aA + bI $$
for some $a, b$. Indeed, this is obvious when $B = pA + qI$. On the other hand, if $A = pB + qI$, then $p \neq 0$ by the assumption, and so, $A+B = \frac{p+1}{p}A - \frac{q}{p}I$.
Next, we consider two possibilities:


*

*If $a = 0$, then $0 = (A+B)^2 = b^2 I$, and so, $b = 0$. This implies $B = -A$.

*If $a \neq 0$, then by rearranging $0 = (A+B)^2$ we obtain $A\left(A + \frac{2b}{a}I\right) = -\frac{b^2}{a^2} I$. Since $A$ is not invertible, this must imply $b = 0$ and hence $B = (a-1)A$.
This completes the proof of Claim. $\square$

Proof of Lemma. For the first part, replacing $A$ by $A-\frac{1}{2}\operatorname{Tr}(A)$ and $B$ by $B-\frac{1}{2}\operatorname{Tr}(B)$ if needed, we may assume that $A$ and $B$ take the form
$$ A = \begin{pmatrix} a & b \\ c & -a \end{pmatrix}, \qquad B = \begin{pmatrix} p & q \\ r & -p \end{pmatrix} $$
Now write $u = (a, p)$, $v = (b, q)$, $w = (c, r)$. Then comparing both sides of $AB - BA = 0$, we find that $AB = BA$ if and only if $\det(u, v) = \det(v, w) = \det(w, u) = 0$. In particular, they are parallel to each other. This implies that $A$ and $B$ are parallel, and so, the desired conclusion follows.
For the second part, we first check $\det(A) = 0$. To this end, assume otherwise. Then
$$ 2 = \operatorname{Tr}(I) = \operatorname{Tr}(A^{-1}A^2A^{-1}) = k \operatorname{Tr}(BA^{-1} - A^{-1}B) = 0,$$
a contradiction. Then taking trace to both sides, by Cayley-Halimton theorem,
$$\operatorname{Tr}(A)^2 = \operatorname{Tr}(A^2) = k \operatorname{Tr}(AB-BA) = 0, $$
and so, $\operatorname{Tr}(A) = 0$. Therefore the desired claim follows from Cayley-Hamilton theorem. $\square$
A: Sangchul Lee's answer can be slightly shortened as follows.
First of all, the key observation is that
$$
(A+B)^2 = 2[B,A] = 2[A+B,A],\tag{1}
$$
where the square bracket denotes a commutator.
By Jacobson's lemma, if $X$ is an $n\times n$ matrix over some field whose characteristic is either zero or $>n$, and $X$ commutes with some commutator $[X,Y]$, the commutator $[X,Y]$ must be nilpotent. So, in our case, if the characteristic of the underlying field is not $2$, since $A+B$ commutes with $(A+B)^2=2[A+B,A]$, Jacobson's lemma implies that $[A+B,A]$ is nilpotent. Hence $(A+B)^2$ and in turn $A+B$ are nilpotent and $(A+B)^2=0$. But then $(1)$ also implies that $[B,A]=0$, i.e. $A$ and $B$ commute.
Let $E$ be the algebraic closure of the field generated by the eight matrix entries in $A$ and $B$. Since $A$ and $B$ commute, they are simultaneously triangulable over $E$. This, together with the facts that $A$ is singular and $A+B$ is nilpotent, imply that $B$ must have a zero eigenvalue, i.e. $B$ is singular.
Let $a=\operatorname{tr}(A)$ and $b=\operatorname{tr}(B)$. On one hand, as both $A$ and $B$ are singular, by Cayley-Hamilton theorem we have
\begin{align}
\operatorname{tr}((A+B)^2)
&=\operatorname{tr}(A^2+2AB+B^2)\\
&=\operatorname{tr}(aA+2AB+bB)\\
&=a^2+2\operatorname{tr}(AB)+b^2.\tag{2}
\end{align}
On the other hand,
\begin{align}
(\operatorname{tr}(A+B))^2
&=(\operatorname{tr}(A)+\operatorname{tr}(B))^2\\
&=a^2+2\operatorname{tr}(A)\operatorname{tr}(B)+b^2.\tag{3}
\end{align}
As $A+B$ is nilpotent, both $\operatorname{tr}((A+B)^2)$ and $(\operatorname{tr}(A+B))^2$ are zero. Now the result follows by equating $(2)$ and $(3)$.
Finally, note that the assumption that characteristic of the field is not $2$ is essential here. E.g. consider
$$
A=\pmatrix{0&0\\ 1&0},\ B=\pmatrix{1&1\\ 0&1}.
$$
If the ground field is $GF(2)$, the conditions that $A^2+3AB+B^2=BA$, or equivalently $(A+B)^2=0$ (because $1+1=0$), and that $\det(A)=0$ are both satisfied, but clearly $\operatorname{tr}(AB)=1\ne0=\operatorname{tr}(A)\operatorname{tr}(B)$ here.
