Matrix Derivative of Log Gaussian with respect to Reparameterized Covariance The traditional multivariate Gaussian density is given by:
$$p(x; \mu, \Sigma) := (2 \pi \det(\Sigma))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (\Sigma)^{-1} (x-\mu)) $$
Suppose I reparameterize the density with another matrix $A$, such that $\Sigma := A^T A$:
$$p(x; \mu, A) := (2 \pi \det(A^T A))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (A^T A)^{-1} (x-\mu)) $$
What is the matrix derivative of the log multivariate Gaussian PDF with respect to $A$? That is, what is
$$ \nabla_A \log p(x; \mu, A)$$
 A: $\def\S{\operatorname{Sym}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$Let
$\,w=(x-\mu),\;$ then the function of interest is
$$\eqalign{
\lambda &= \log(p) 
  \;=\; -\tfrac 12 w^T\Sigma^{-1}w - \tfrac 12\log\det(\Sigma) 
  - \tfrac 12\log(2\pi) \\
}$$
First expand the differential in terms of $\Sigma$
$$\eqalign{
d\lambda
 &= \tfrac 12\Sigma^{-1}ww^T\Sigma^{-1}:d\Sigma - \tfrac 12\Sigma^{-1}:d\Sigma \\
}$$
Then substitute $\;\Sigma = A^TA$
$$\eqalign{
d\Sigma &= dA^TA+A^TdA \\&= 2\operatorname{Sym}(A^TdA) \\\\
d\lambda
 &= \Big(\tfrac 12\Sigma^{-1}ww^T\Sigma^{-1} - \tfrac 12\Sigma^{-1}\Big)
  : 2\operatorname{Sym}(A^TdA) \\
 &= \operatorname{Sym}\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big)
  : \Big(A^TdA\Big) \\
 &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big)
  : dA \\\\
\p{\lambda}{A}
 &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) \\\\
}$$
If you wish, you can eliminate the remaining $\Sigma$ variables in favor of $A$
$$\eqalign{
A\Sigma^{-1} &= A\left(A^TA\right)^{-1} = \left(A^+\right)^T \\
\Sigma^{-1} &= A^+\left(A^+\right)^T \\
}$$
where $A^+$ denotes the Moore-Penrose inverse of $A$.

In some steps above, a colon is used to denote the trace/Frobenius product, i.e.
$$\eqalign{
A:B &= {\rm Tr}(A^TB) \\
A:A &= \big\|A\big\|_F^2 \\
}$$
