Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this community
The traditional multivariate Gaussian density is given by:
$$p(x; \mu, \Sigma) := (2 \pi \det(\Sigma))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (\Sigma)^{-1} (x-\mu)) $$
Suppose I reparameterize the density with another matrix $A$, such that $\Sigma := A^T A$:
$$p(x; \mu, A) := (2 \pi \det(A^T A))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (A^T A)^{-1} (x-\mu)) $$
What is the matrix derivative of the log multivariate Gaussian PDF with respect to $A$? That is, what is
$$ \nabla_A \log p(x; \mu, A)$$
$\def\S{\operatorname{Sym}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$Let
$\,w=(x-\mu),\;$ then the function of interest is
$$\eqalign{
\lambda &= \log(p)
\;=\; -\tfrac 12 w^T\Sigma^{-1}w - \tfrac 12\log\det(\Sigma)
- \tfrac 12\log(2\pi) \\
}$$
First expand the differential in terms of $\Sigma$
$$\eqalign{
d\lambda
&= \tfrac 12\Sigma^{-1}ww^T\Sigma^{-1}:d\Sigma - \tfrac 12\Sigma^{-1}:d\Sigma \\
}$$
Then substitute $\;\Sigma = A^TA$
$$\eqalign{
d\Sigma &= dA^TA+A^TdA \\&= 2\operatorname{Sym}(A^TdA) \\\\
d\lambda
&= \Big(\tfrac 12\Sigma^{-1}ww^T\Sigma^{-1} - \tfrac 12\Sigma^{-1}\Big)
: 2\operatorname{Sym}(A^TdA) \\
&= \operatorname{Sym}\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big)
: \Big(A^TdA\Big) \\
&= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big)
: dA \\\\
\p{\lambda}{A}
&= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) \\\\
}$$
If you wish, you can eliminate the remaining $\Sigma$ variables in favor of $A$
$$\eqalign{
A\Sigma^{-1} &= A\left(A^TA\right)^{-1} = \left(A^+\right)^T \\
\Sigma^{-1} &= A^+\left(A^+\right)^T \\
}$$
where $A^+$ denotes the Moore-Penrose inverse of $A$.
In some steps above, a colon is used to denote the trace/Frobenius product, i.e. $$\eqalign{ A:B &= {\rm Tr}(A^TB) \\ A:A &= \big\|A\big\|_F^2 \\ }$$
