# Matrix Derivative of Log Gaussian with respect to Reparameterized Covariance

The traditional multivariate Gaussian density is given by:

$$p(x; \mu, \Sigma) := (2 \pi \det(\Sigma))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (\Sigma)^{-1} (x-\mu))$$

Suppose I reparameterize the density with another matrix $$A$$, such that $$\Sigma := A^T A$$:

$$p(x; \mu, A) := (2 \pi \det(A^T A))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (A^T A)^{-1} (x-\mu))$$

What is the matrix derivative of the log multivariate Gaussian PDF with respect to $$A$$? That is, what is

$$\nabla_A \log p(x; \mu, A)$$

$$\def\S{\operatorname{Sym}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$$Let $$\,w=(x-\mu),\;$$ then the function of interest is \eqalign{ \lambda &= \log(p) \;=\; -\tfrac 12 w^T\Sigma^{-1}w - \tfrac 12\log\det(\Sigma) - \tfrac 12\log(2\pi) \\ } First expand the differential in terms of $$\Sigma$$ \eqalign{ d\lambda &= \tfrac 12\Sigma^{-1}ww^T\Sigma^{-1}:d\Sigma - \tfrac 12\Sigma^{-1}:d\Sigma \\ } Then substitute $$\;\Sigma = A^TA$$ \eqalign{ d\Sigma &= dA^TA+A^TdA \\&= 2\operatorname{Sym}(A^TdA) \\\\ d\lambda &= \Big(\tfrac 12\Sigma^{-1}ww^T\Sigma^{-1} - \tfrac 12\Sigma^{-1}\Big) : 2\operatorname{Sym}(A^TdA) \\ &= \operatorname{Sym}\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) : \Big(A^TdA\Big) \\ &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) : dA \\\\ \p{\lambda}{A} &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) \\\\ } If you wish, you can eliminate the remaining $$\Sigma$$ variables in favor of $$A$$
\eqalign{ A\Sigma^{-1} &= A\left(A^TA\right)^{-1} = \left(A^+\right)^T \\ \Sigma^{-1} &= A^+\left(A^+\right)^T \\ } where $$A^+$$ denotes the Moore-Penrose inverse of $$A$$.
In some steps above, a colon is used to denote the trace/Frobenius product, i.e. \eqalign{ A:B &= {\rm Tr}(A^TB) \\ A:A &= \big\|A\big\|_F^2 \\ }
• One quick question: what happened to the $\frac{1}{2}$ in front of $w^T \Sigma^{-1}w$ when moving from $\lambda$ to $d\lambda$? Mar 26, 2021 at 15:33
• The missing $\tfrac 12$ was a mistake, which has been fixed. The standard text for the subject is probably Matrix Differential Calculus (by Magnus and Neudecker), but there are a lot of interesting posts on this site under the "matrix-calculus" tag.