2
$\begingroup$

The traditional multivariate Gaussian density is given by:

$$p(x; \mu, \Sigma) := (2 \pi \det(\Sigma))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (\Sigma)^{-1} (x-\mu)) $$

Suppose I reparameterize the density with another matrix $A$, such that $\Sigma := A^T A$:

$$p(x; \mu, A) := (2 \pi \det(A^T A))^{-1/2} \exp(-\frac{1}{2} (x-\mu)^T (A^T A)^{-1} (x-\mu)) $$

What is the matrix derivative of the log multivariate Gaussian PDF with respect to $A$? That is, what is

$$ \nabla_A \log p(x; \mu, A)$$

$\endgroup$

1 Answer 1

2
$\begingroup$

$\def\S{\operatorname{Sym}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$Let $\,w=(x-\mu),\;$ then the function of interest is $$\eqalign{ \lambda &= \log(p) \;=\; -\tfrac 12 w^T\Sigma^{-1}w - \tfrac 12\log\det(\Sigma) - \tfrac 12\log(2\pi) \\ }$$ First expand the differential in terms of $\Sigma$ $$\eqalign{ d\lambda &= \tfrac 12\Sigma^{-1}ww^T\Sigma^{-1}:d\Sigma - \tfrac 12\Sigma^{-1}:d\Sigma \\ }$$ Then substitute $\;\Sigma = A^TA$ $$\eqalign{ d\Sigma &= dA^TA+A^TdA \\&= 2\operatorname{Sym}(A^TdA) \\\\ d\lambda &= \Big(\tfrac 12\Sigma^{-1}ww^T\Sigma^{-1} - \tfrac 12\Sigma^{-1}\Big) : 2\operatorname{Sym}(A^TdA) \\ &= \operatorname{Sym}\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) : \Big(A^TdA\Big) \\ &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) : dA \\\\ \p{\lambda}{A} &= A\,\Big(\Sigma^{-1}ww^T\Sigma^{-1} - \Sigma^{-1}\Big) \\\\ }$$ If you wish, you can eliminate the remaining $\Sigma$ variables in favor of $A$
$$\eqalign{ A\Sigma^{-1} &= A\left(A^TA\right)^{-1} = \left(A^+\right)^T \\ \Sigma^{-1} &= A^+\left(A^+\right)^T \\ }$$ where $A^+$ denotes the Moore-Penrose inverse of $A$.


In some steps above, a colon is used to denote the trace/Frobenius product, i.e. $$\eqalign{ A:B &= {\rm Tr}(A^TB) \\ A:A &= \big\|A\big\|_F^2 \\ }$$

$\endgroup$
4
  • $\begingroup$ One quick question: what happened to the $\frac{1}{2}$ in front of $w^T \Sigma^{-1}w$ when moving from $\lambda$ to $d\lambda$? $\endgroup$ Commented Mar 26, 2021 at 15:33
  • $\begingroup$ Another quick question: could you suggest textbooks/references with good practice problems to help me improve at matrix calculus using this sort of notation? $\endgroup$ Commented Mar 26, 2021 at 15:40
  • 1
    $\begingroup$ The missing $\tfrac 12$ was a mistake, which has been fixed. The standard text for the subject is probably Matrix Differential Calculus (by Magnus and Neudecker), but there are a lot of interesting posts on this site under the "matrix-calculus" tag. $\endgroup$
    – greg
    Commented Mar 26, 2021 at 17:39
  • $\begingroup$ Wonderful. Thank you! $\endgroup$ Commented Mar 26, 2021 at 17:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .