Derivative of trace of fourth-order matrix product? I've been able to find all I need in the handy Matrix Cookbook, but today I encountered a form that is not in the book. Could anyone show me what is
$$\frac{\operatorname{d}}{\operatorname{d}X}\operatorname{tr}(A^TX^TXAB^TX^TXB)$$

MATLAB's symbolic computations told me
>> syms X A B;
>> diff(trace(A'*X'*X*A*B'*X'*X*B), X)

ans =

2*A*B*X*conj(A)*conj(B)*conj(X)^2 + 2*A*B*X^2*conj(A)*conj(B)*conj(X)

But a quick check shows that I am using MATLAB wrongly,
>> syms X A;
>> diff(trace(X*A), X)

ans =

A

since the correct answer should be $A^T$.
 A: Let $f(X)=\operatorname{tr}(A^TX^TXAB^TX^TXB)$. Then the first order approximation of $f(X+H)-f(X)$ (when $H$ is small) is given by
\begin{align}
&\phantom{=}\operatorname{tr}(A^TH^TXAB^TX^TXB)
+\operatorname{tr}(A^TX^TH\color{red}{AB^TX^TXB})\\
&\phantom{+}+\operatorname{tr}(A^TX^TXAB^TH^TXB)
+\operatorname{tr}(A^TX^TXAB^TX^TH\color{red}{B})\\
&=\operatorname{tr}(H^TXAB^TX^TXBA^T)
+\operatorname{tr}(\color{red}{AB^TX^TXB}A^TX^TH)\\
&\phantom{+}+\operatorname{tr}(H^TXBA^TX^TXAB^T)
+\operatorname{tr}(\color{red}{B}A^TX^TXAB^TX^TH)\\
&=2\operatorname{tr}(H^TXAB^TX^TXBA^T)
+2\operatorname{tr}(H^TXBA^TX^TXAB^T),
\end{align}
where we have used the property that $\operatorname{tr}(PQ)=\operatorname{tr}(QP)$ in the first equality and that $\operatorname{tr}(P)=\operatorname{tr}(P^T)$ in the second one. It follows that $\frac{df(X)}{dX}=2(XAB^TX^TXBA^T+XBA^TX^TXAB^T)$. (And of course, if you use a transposed layout for the derivative, you should take the transpose of the above expression as the answer.)
A: Using product rule for differential forms, cyclic permutations, and that trace is linear 
$$dtr(A^TX^TXAB^TX^TXB)=dtr(BA^TX^TXAB^TX^TX)=tr(d(BA^TX^TXAB^TX^TX))$$
$$=tr(d(BA^TX^T)XAB^TX^TX+BA^TX^Td(XAB^TX^TX))$$
$$=tr(d(BA^TX^T)XAB^TX^TX+BA^TX^T(d(XAB^T)X^TX+XAB^Td(X^TX)))$$
$$=tr(d(BA^TX^T)XAB^TX^TX+BA^TX^T(d(XAB^T)X^TX+XAB^T(d(X^T)X+X^Td(X)))$$
$$=tr(d(BA^TX^T)XAB^TX^TX+BA^TX^Td(XAB^T)X^TX+BA^TX^TXAB^T(d(X^T)X+X^Td(X)))$$
$$=tr(d(BA^TX^T)XAB^TX^TX+BA^TX^Td(XAB^T)X^TX+BA^TX^TXAB^Td(X^T)X+BA^TX^TXAB^TX^Td(X))$$
$$=tr(BA^Td(X^T)XAB^TX^TX)+tr(BA^TX^Td(X)AB^TX^TX)+tr(BA^TX^TXAB^Td(X^T)X)+tr(BA^TX^TXAB^TX^Td(X))$$
$$=tr((BA^Td(X^T)XAB^TX^TX)^T)+tr(BA^TX^Td(X)AB^TX^TX)+tr((BA^TX^TXAB^Td(X^T)X)^T)+tr(BA^TX^TXAB^TX^Td(X))$$
$$=tr(X^TXBA^TX^T(dX)AB^T)+tr(BA^TX^Td(X)AB^TX^TX)+tr(X^T(dX)BA^TX^TXAB^T)+tr(BA^TX^TXAB^TX^Td(X))$$
$$=tr(AB^TX^TXBA^TX^T(dX))+tr(AB^TX^TXBA^TX^Td(X))+tr(BA^TX^TXAB^TX^T(dX))+tr(BA^TX^TXAB^TX^Td(X))$$
$$=tr(AB^TX^TXBA^TX^T(dX)+AB^TX^TXBA^TX^T(dX)+BA^TX^TXAB^TX^T(dX)+BA^TX^TXAB^TX^Td(X))$$
$$=tr((AB^TX^TXBA^TX^T+AB^TX^TXBA^TX^T+BA^TX^TXAB^TX^T+BA^TX^TXAB^TX^T)d(X))$$
$$=tr((2AB^TX^TXBA^TX^T+2BA^TX^TXAB^TX^T)d(X))$$
this shows that $\frac{\partial tr(A^TX^TXAB^TX^TXB)}{\partial X}= 2AB^TX^TXBA^TX^T+2BA^TX^TXAB^TX^T$
A: For convenience, let
$$\eqalign{
 C &= AB^T \cr
 Y &= X^TX = Y^T \cr
}$$
Use the Frobenius (:) Inner Product and these new variables to write the function, differential, and gradient as
$$\eqalign{
 f &= C:YCY \cr\cr
df &= C:dY\,CY + C:YC\,dY \cr
   &= CYC^T:dY + C^TYC:dY \cr
   &= (CYC^T+C^TYC):dY \cr
   &= (CYC^T+C^TYC):2\,{\rm sym}(X^TdX) \cr
   &= 2\,{\rm sym}(CYC^T+C^TYC):X^TdX \cr
   &= 2\,X\,(CYC^T+C^TYC):dX \cr\cr
\frac{\partial f}{\partial X} &= 2\,X\,(CYC^T+C^TYC) \cr
   &= 2\,X\,(AB^TX^TXBA^T+BA^TX^TXAB^T) \cr
}$$
