Show trace is zero Problem: We are given $n\times n$ square matrices $A$ and $B$ with $AB+BA=0$ and $A^2+B^2=I$. Show $tr(A)=tr(B)=0$.
Thoughts: We have $tr(BA)=tr(AB)=-tr(BA)=0$. We also have the factorizations $(A+B)^2=I$ and $(A-B)^2=I$ by combining the two relations above.
Let $\alpha_i$ denote the eigenvalues of $A$, and $\beta_i$ the eigenvalues of $B$. We have, by basic properties of trace,
$\sum \alpha_i^2 +\sum \beta_i^2=n$
from $A^2+B^2=I$. 
I'm not sure where to go from here.
I would prefer a small hint to a complete answer.
 A: It appears from the context in the book that the correct problem is 
$$ A^2 + B^2 = A B + B A = 0.    $$
 The middle step is that $(B-A)^2 = 0,$ so we name the nilpotent matrix $N=B-A.$  Wait, I think that is enough. Because it is also true that $(A+B)^2 = 0.$ So $A+B$ and $B-A$ both have trace $0.$ So $tr \; \; 2B = 0.$ That finishes characteristic other than 2. We don't need full Jordan form for nilpotent matrices, just a quick proof that $N^2 = 0$ implies that the trace of $N$ is zero. Hmmm. This certainly does follow from the fact that a nilpotent matrix over any field has a Jordan form, but I cannot say that I have seen a proof of that. 
Alright, in characteristic 2 this does not work, in any dimension take
$$ A = B,  $$
$$  A =  B   \; \; \; \mbox{then} \; \; A^2 + B^2 = 2 A^2 = 0, \; AB + BA = 2 A^2 = 0. $$
In comparison, the alternate problem 
$$ A^2 + B^2 = A B + B A = I    $$ has the same thing about nilpotence, however in fields where $2 \neq 0$ and $2$ is a square we get a counterexample with
$$ A \; = \;  
 \left(  \begin{array}{rr}
  \frac{1}{\sqrt 2} & \frac{-1}{2}  \\
   0  &   \frac{1}{\sqrt 2} 
\end{array} 
  \right)  
  $$
and
$$ B \; = \;  
 \left(  \begin{array}{rr}
  \frac{1}{\sqrt 2} & \frac{1}{2}  \\
   0  &   \frac{1}{\sqrt 2} 
\end{array} 
  \right)  
  $$
A: I will prove this just for the case $2\times 2$ being the approach similar for $n\times n$. Any matrices can be written through the generators of U(2) group as
$$
   A=a_0I+a_1\sigma_1+a_2\sigma_2+a_3\sigma_3=a_0I+{\bf a}\cdot{\bf \sigma}
$$
$$
   B=b_0I+b_1\sigma_1+b_2\sigma_2+b_3\sigma_3=b_0I+{\bf b}\cdot{\bf \sigma}
$$
having $tr(\sigma_i)=0$, $\sigma_i\sigma_j+\sigma_j\sigma_i=2\delta_{ij}I$, $\sigma_i\sigma_j-\sigma_j\sigma_i=2i\epsilon_{ijk}\sigma_k$ and $\sigma_i^2=I$. Besides, we assume $det A, det B\ne 0$. Now we have
$$
  AB+BA=2a_0b_0+2(a_0{\bf b}\cdot{\bf \sigma}+b_0{\bf a}\cdot{\bf \sigma})+2{\bf a}\cdot{\bf b}=0.
$$
and so, it must be $a_0=b_0=0$ and ${\bf a}\cdot{\bf b}=0$. Similarly, one has
$$
  A^2+B^2=(|{\bf a}|^2+|{\bf b}|^2)I
$$
and then $|{\bf a}|^2+|{\bf b}|^2=1$. This implies $tr(A)=tr(B)=0$. These matrices are orthogonal in some sense.
