Let A,B be two $4\times4$ real commuting matrices and $\det(A^2-AB+B^2) = 0$ show that $\det(A+B) + 3\det(A-B) = 6\det(A) + 6\det(B)$ Let $A$, $B$ be two $4\times 4$ matrices with real enteries, such that $AB = BA$ and $\det(A^2-AB+B^2) = 0$. Show that $\det(A+B) + 3\det(A-B) = 6\det(A) + 6\det(B)$.
How can I get started on this question? 
 A: We can exploit the fact that each expression is homogeneous on $A$ and $B$.
Assume that $\det(B) \neq 0$. Multiply both sides of the first equation by $\det(B)^{-2}$, and both sides of the second equation by $\det(B)^{-1}$.  Using the fact that $A$ and $B$ commute, we have
$$
\det(A^2 - AB + B^2)\det(B)^{-2} = 0 \implies\\
\det(B^{-2}(A^2 - AB + B^2)) = 0 \implies\\
\det((AB^{-1})^2 - (AB^{-1}) + I) = 0.
$$
Similarly, multiply both sides of the second equation by $\det(B)^{-1}$ to get
$$
\det((AB^{-1}) + I) + 3 \det((AB^{-1}) - I) = 6 \det(AB^{-1}) + 6.
$$
So, let $X = AB^{-1}$. It suffices to show that 
$$
\det(X^2 - X + I) = 0 \implies \det(X + I) + 3 \det(X - I) = 6 \det(X) + 6.
$$
From there, the trick is hidden below (put your mouse over it to see).  This is enough to get started though, and I recommend that you try to figure this part out for yourself.
Hint:

 $X$ is a matrix with real entries, so its complex eigenvalues must come in conjugate pairs.  

Solution:

Thus, we can guarantee that the roots of $x^2 - x + 1 = 0$, 
 namely $\lambda_\pm = \frac 12 \pm \frac{\sqrt{3}}2 i$, must be eigenvalues of 
 $X$.  Let $a,b$ denote the remaining two eigenvalues.  We can now rewrite each 
 expression as follows:

 $$ \det(X+I) = (\lambda_+ +1)(\lambda_- + 1)(a+1)(b+1) = 3(a+1)(b+1)\\ \det(X-I) = (\lambda_+ - 1)(\lambda_- - 1)(a-1)(b-1) = (a-1)(b-1)\\ \det(X) = \lambda_+ \lambda_- ab = ab$$

 We can now show by routine simplification that the desired equality holds in the case that $\det(B) \neq 0$.

For the case that $\det(B) = 0$, it suffices to makes an argument by continuity.  Since 
$$
\det(A+(B + \epsilon I)) + 3\det(A-(B + \epsilon I)) = 6\det(A) + 6\det(B + \epsilon I)
$$
will hold for all sufficiently small $\epsilon > 0$ (which will force $B + \epsilon I$ to be invertible), it must also hold that the two sides are equal for $\epsilon = 0$.
A: If $\det(A^2-AB+B^2)=0$, 
as one may see by simultaneous trigonalisation $AB=BA$, that some couple of eigenvalues $\lambda_i^2(A)-\lambda_i(B)\lambda_i(A)+\lambda_i^2(B)=0$
The case $\lambda_i(A)=\lambda_i(B)=0$ is easy.
 For $\lambda_i(B)=c\neq 0$ fixed, $\lambda_i(A)=c\dfrac{1\pm\sqrt{3}I}{2}=a$, as $A$ commutes with $B$ again there is a changement of basis for that $a_{1,1}=a$ and $a_{2,2}=\bar{a}$, on the same line i.e for the same eigenvector $b_{1,1}=c$ and $b_{2,2}=\bar{c}$. The rest is a computation exercise for those (upper triangular) $A$ and $B$.
