Derivative of $\operatorname{Tr}(AX(X^TX)^{-1}X^T)$ w.r.t. $X$. I have a very complex trace function $F(X)=\operatorname{Tr}(AX(X^TX)^{-1}X^T)$. I would like to find the derivative $\nabla_{X}F(X)$. I do not know how to do it. I checked the matrix cookbook there is nothing similar to this. Please help!
 A: My usual approach to solving all such types of derivatives is with the differential. Let $f(X) = \operatorname{tr}(AX(X^TX)^{-1}X)$ and let $\delta X$ be an arbitrary variation of $X$. Then, by linearity of trace and the product rule,
$$df(X)\delta X = \operatorname{tr}\left(A\delta X (X^TX)^{-1}X + AXd[(X^TX)^{-1}]\delta X X + AX(X^TX)^{-1} \delta X\right).$$
To evaluate the middle derivative we can use the formula for the differential of matrix inverse,
$$d\left[X^{-1}\right]\delta X = -X^{-1} \delta X X^{-1},$$
and more product rules:
$$df(X)\delta X = \operatorname{tr}\left(A\delta X (X^TX)^{-1}X - AX(X^TX)^{-1} \delta X^T X(X^TX)^{-1} X - AX(X^TX)^{-1} X^T \delta X (X^TX)^{-1} X + AX(X^TX)^{-1} \delta X\right).$$
This formula can be cleaned up using invariance of trace to cyclic permutations:
$$df(X)\delta X = \operatorname{tr}\left((X^TX)^{-1}X A\delta X  - X(X^TX)^{-1} X AX(X^TX)^{-1} \delta X^T  - (X^TX)^{-1} X  AX(X^TX)^{-1} X^T \delta X + AX(X^TX)^{-1} \delta X\right),$$
or
$$df(X)\delta X = \left[A^T X ^T (X^TX)^{-1}  - X(X^TX)^{-1} (X AX + X^TA^TX^T)(X^TX)^{-1} + (X^TX)^{-1} X^T A^T\right] : \delta X,$$
with more simplifications possible if more information is known about $A$ and $X$: if they're symmetric, for instance, or if you have a typo and the last $X$ is supposed to be $X^T$; or if $X$ is square and invertible.
A: $
\def\l{\lambda}\def\o{{\tt1}}\def\p{\partial}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\diag#1{\operatorname{diag}\LR{#1}}
\def\Diag#1{\operatorname{Diag}\LR{#1}}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
\def\c#1{\color{red}{#1}}
\def\CLR#1{\c{\LR{#1}}}
\def\gradLR#1#2{\LR{\grad{#1}{#2}}}
$For typing convenience, define the matrix variable
$$\eqalign{
M &= \LR{X^TX}^{-1} \qiq
dM &= -M\LR{dX^TX+X^TdX}M \\
}$$
and use a colon to denote the Frobenius product,
which is a concise notation for the trace
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\
A:A &= \|A\|^2_F \\
}$$
The properties of the underlying trace function allow the terms in a
Frobenius product to be rearranged in many different ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
C:\LR{AB} &= \LR{CB^T}:A \\&= \LR{A^TC}:B \\
}$$

Write the cost function using the above notation,
and calculate its differential and gradient.
$$\eqalign{
F &= A:XMX^T \\
dF
 &= A:\LR{dX\,MX^T+X\,dM\,X^T+XM\,dX^T} \\
 &= AXM:dX + X^TAX:\c{dM} + A^TXM:dX \\
 &= \LR{A+A^T}XM:dX \c{-} X^TAX:\CLR{M\LR{dX^TX+X^TdX}M} \\
 &= \LR{A+A^T}XM:dX - XMX^T\LR{A+A^T}XM:dX \\
 &= \LR{I-XMX^T}\LR{A+A^T}XM:dX \\
\grad{F}{X}
 &= \LR{I-XMX^T}\LR{A+A^T}XM \\
}$$
