Compute $\det(A^n+B^n)$ 
Let $A, B $ be two real $3\times 3 $ matrices, $AB=BA$, and $ \det(A-B)=\det(A^2+B^2)=1,\det(A+B)=3, \det(B)=0 $, then, what is ? 

$$\det(A^n+B^n)$$

here $n$ is a positive integer. 


The problem looks very horrible. Any help will be appreciated
 A: Although I don't know the real tricks behind the problem, the problem is not as terrible as it appears.
As $A$ and $B$ commute, they are simultanesouly triangularisable over $\mathbb C$. Given that $\det(B)=0$, we may assume that $A$ and $B$ are triangular matrices whose diagonal entries are respectively $a_1,a_2,a_3$ and $b_1,b_2,0$. So, the other three determinant conditions amount to
\begin{align}
(a_1-b_1)(a_2-b_2)a_3&=1,\tag{1}\\
(a_1+b_1)(a_2+b_2)a_3&=3,\tag{2}\\
(a_1^2+b_1^2)(a_2^2+b_2^2)a_3^2&=1.\tag{3}
\end{align}
Hence $a_3$ is nonzero. Now $A$ must be invertible. Otherwise, if, say, $a_2=0$, then the sum and difference of $(1)$ and $(2)$ would give $a_1b_2a_3=1$ and $b_1b_2a_3=2$, which contradict $(3)$.
Let $C=A^{-1}B$. Then $C$ is singular (because $B$ is). Let its complex eigenvalues be $x,y,0$. By the given conditions, we have
\begin{align}
\det(I+C)&=3\det(I-C),\\
(1+x)(1+y)&=3(1-x)(1-y),\\
2(x+y)&= xy + 1,\tag{4}
\end{align}
and also
\begin{align}
\det(I-C)^2&=\det(I+C^2),\\
(1-x)^2(1-y)^2 &= (1+x^2)(1+y^2),\\
4xy - 2(x+y)(xy+1) &= 0,\\
4xy - (xy+1)^2 &= 0 \quad\text{ by } (4),\\
xy &= 1.\tag{5}
\end{align}
Now equations $(4)$ and $(5)$ show that $x$ and $y$ are roots of the equations $xy=1$ and $x+y=1$. As $C$ is a real matrix, nonreal eigenvalues must occur in a conjugate pair, so $\{x,y\}=\{e^{i\pi/3},e^{-i\pi/3}\}$. Substitute $b_1=e^{i\pi/3}a_1$ and $b_2=e^{-i\pi/3}a_2$ into $(1)$, we get $\det(A)=1$. Therefore
$$
\det(A^n+B^n)=\det(I+C^n)\det(A)^n=|1+e^{in\pi/3}|^2=2 + 2\cos\left(\frac{n\pi}{3}\right).
$$
