$trace (A^2)=trace(AA') \iff A= A'$ Let A be an $n\times n$ matrix then,
$$trace (A^2)=trace(AA') \iff A= A'$$
where $A'$ is the transpose of $A$.
only if part is easy by observing that $A=A'$ implies that $A^2=AA'$ and trace of $A^2-AA'$ is zero, since all the entries are zero.
But the if part is ....
 A: You sure this is correct for general $A$?
Let $A$ be a matrix such that $tr(A)=0$.
Let $$A'=A+I$$
Then $$tr(AA')=tr(A^2)$$
but $$A \ne A'$$
ADDED:
Let $$B=\frac{A+A^T}{2}$$
$$C=\frac{A-A^T}{2}$$
$$tr(A^2)=tr(AA^T)$$
$$tr((B+C)(B+C))=tr((B+C)(B-C))$$
$$tr(B^2+BC+CB+C^2)=tr(B^2-BC+CB-C^2)$$
$$tr(BC+C^2)=tr(-BC-C^2)=-tr((B+C)C)$$
$$=-tr(C(B+C))=-tr(BC+C^2)$$
Hence $$tr(BC+C^2)=tr(C^2)=0$$
$$tr(CC^T)=0$$
$$\sum_{ij}C_{ij}^2=0$$
$$C_{ij}=0$$ for all $i,j$.
Hence, $$A=A^T$$.
Note: I have assumed real matrices. I fancy your $A'$ is in fact meant to be the Hermitian adjoint $A^\dagger$， for complex $A$ in general.
A: Yes, the if part was easy ^^
Try to use the Cauchy Schwarz inequality to see what you get.
Or more specifically, the Cauchy Schwarz equality 
$$Tr(A^2) = \langle A\mid A^*\rangle  \leq \sqrt{\langle A\mid A\rangle  \langle A^*\mid A^*\rangle} = Tr(AA^*)\quad \text{ by C-S}$$
And you see that you have equality in CS if $tr(AA′)=tr(A^2)$, $A$ and $A^*$ must be collinear and you can conclude.
A: You'll want to try and prove that $A-A^t=0$. This can be done by proving $\Vert A-A^t\Vert_2=0$.
Then :
$$\Vert A-A^t\Vert_2^2=tr((A-A^t)(A-A^t)^t)=tr(AA^t-A^2+A^tA-(A^t)^2)$$
$$=tr(AA^t-A^2)+tr(A^tA-(A^t)^2)=0+tr(AA^t)-tr(A^2)=0$$
The key properties used here are :


*

*$tr(XX^t)=\sum_{i,j}X_{ij}^2=\Vert X\Vert_2^2$

*linearity of $tr$

*$tr(X^t)=tr(X)$

*$tr(XY)=tr(YX)$ 

A: There is a result,
$tr(B'B)=0 \iff B =0$, this result is easy to prove.
Now in your case consider $B=A-A'$
$$
tr((A-A')'(A-A'))= tr(A'A)-tr(A'^2) - tr(A^2) + tr(AA') =0
$$
because $tr(A^2)= tr(AA')$ and $tr(A'^2)=tr(A'A)$
Hence the result, $A-A'=0$
