# Derivation of derivative of multivariate Gaussian w.r.t. covariance matrix

I'm reading a paper, probabilistic CCA, in which the authors state derivatives without showing derivations. I would like step-by-step derivations to convince myself. Consider a $d$-dimensional multivariate Gaussian random variable:

$$\textbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)$$

In probabilistic CCA, we define $\Sigma = W W^{\top} + \Psi$, where $W \in \mathbb{R}^{d \times q}$ and $\Psi \in \mathbb{R}^{d \times d}$. I'd like to compute the derivative w.r.t. $\boldsymbol{\mu}$, $W$, and $\Psi$ for the negative log-likelihood.

The stationary point for $\boldsymbol{\mu}$ is just the empirical mean (shown below*) or $\hat{\boldsymbol{\mu}}$. Plugging in the minimum for the parameter $\boldsymbol{\mu}$ into the negative log-likelihood, we get:

$$\frac{\partial \mathcal{L}}{\partial W} = \frac{\partial}{\partial W} \Big\{ \overbrace{ \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \hat{\boldsymbol{\mu}})^{\top} \Sigma^{-1} (\textbf{x}_i - \hat{\boldsymbol{\mu}}) }^{A} + \overbrace{\frac{n}{2} \ln |\Sigma|}^{B} + \overbrace{\text{const}}^{C} \Big\}$$

Clearly, $C = 0$. But I'm not sure how to handle $A$ and $B$, particularly since $\Sigma = W W^{\top} + \Psi$.

### *Derivative w.r.t. $\boldsymbol{\mu}$

The negative log-likelihood is:

$$\mathcal{L} = \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) + \frac{n}{2} \ln |\Sigma| + \text{const}$$

The derivative of the two rightmost terms with respect to $\boldsymbol{\mu}$ is $0$, meaning we just need to compute:

$$\frac{\partial}{\partial \boldsymbol{\mu}} \Big\{ \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0$$

By the linearity of differentiation, we have:

$$\frac{1}{2} \sum_{i=1}^{n} \frac{\partial}{\partial \boldsymbol{\mu}} \Big\{ (\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0$$

Using Equation ($86$) from the Matrix Cookbox, we get:

$$\frac{1}{2} \sum_{i=1}^{n} \Big\{ -2 \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0$$

Finally, solve for $\boldsymbol{\mu}$, we get:

\begin{align} 0 &= \frac{1}{2} \sum_{i=1}^{n} \Big\{ -2 \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} \\ &= - \sum_{i=1}^{n} \Big\{ \Sigma^{-1} \textbf{x}_i - \Sigma^{-1} \boldsymbol{\mu} \Big\} \\ &= - \sum_{i=1}^{n} \Big\{ \Sigma^{-1} \textbf{x}_i \Big\} + n \Sigma^{-1} \boldsymbol{\mu} \\ - n \Sigma^{-1} \boldsymbol{\mu} &= - \Sigma^{-1} \sum_{i=1}^{n} \textbf{x}_i \\ \boldsymbol{\mu} &= \frac{1}{n} \sum_{i=1}^{n} \textbf{x}_i \end{align}

And we're done.

All those Greek letters are a pain to type, so let's use these variables \eqalign{ S = \Sigma,\,\,\,P = \Phi,\,\,\,L={\mathcal L},\,\,\,Z = (X-\mu 1) \cr } where $X$ is the matrix whose columns are the $x_i$ vectors, and $(\mu 1)$ is a matrix all of whose elements are equal to $\mu$.
Further, let's use a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Write the objective function in terms of the Frobenius product and these new variables. Then find its differential and gradients. \eqalign{ L &= \tfrac{n}{2}\log(\det(S)) + \tfrac{1}{2}ZZ^T:S^{-1} + K \cr dL &= \tfrac{n}{2}{\rm tr\,}(d\log(S)) + \tfrac{1}{2}ZZ^T:dS^{-1} + 0 \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):dS \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):d(WW^T+P) \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):(dW\,W^T+ W\,dW^T+dP) \cr } Setting $dW=0$ yields the gradient wrt $P$ \eqalign{ dL &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):dP \cr \frac{\partial L}{\partial P} &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)\cr } While setting $dP=0$ recovers the gradient wrt $W$ \eqalign{ dL &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):(dW\,W^T+ W\,dW^T) \cr &= \Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)W:dW \cr \frac{\partial L}{\partial W} &= \Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)W \cr } In several of the steps, we've made use of the fact that $S$ is symmetric.
• Thanks! I can follow your derivation of $B$, the log of the determinant. I'm confused about how you setup for $A$ though. My part $A$ is $\frac{1}{2} \sum z^{\top} S z$. If I use the trace trick, that's: $\frac{1}{2} \sum \text{tr}(z z^{\top} S)$. But your setup is $\frac{1}{2} \text{tr}(z^{\top} z S^{-1})$. What happened to the summation? – gwg Jul 30 '18 at 19:47
• Sorry, I misread the A term. The answer has been updated with the correct term. The change consisted of changing the $z$ vector into the $Z$ matrix. – greg Jul 30 '18 at 20:36