I'm reading a paper, probabilistic CCA, in which the authors state derivatives without showing derivations. I would like step-by-step derivations to convince myself. Consider a $d$-dimensional multivariate Gaussian random variable:

$$ \textbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma) $$

In probabilistic CCA, we define $\Sigma = W W^{\top} + \Psi$, where $W \in \mathbb{R}^{d \times q}$ and $\Psi \in \mathbb{R}^{d \times d}$. I'd like to compute the derivative w.r.t. $\boldsymbol{\mu}$, $W$, and $\Psi$ for the negative log-likelihood.

The stationary point for $\boldsymbol{\mu}$ is just the empirical mean (shown below*) or $\hat{\boldsymbol{\mu}}$. Plugging in the minimum for the parameter $\boldsymbol{\mu}$ into the negative log-likelihood, we get:

$$ \frac{\partial \mathcal{L}}{\partial W} = \frac{\partial}{\partial W} \Big\{ \overbrace{ \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \hat{\boldsymbol{\mu}})^{\top} \Sigma^{-1} (\textbf{x}_i - \hat{\boldsymbol{\mu}}) }^{A} + \overbrace{\frac{n}{2} \ln |\Sigma|}^{B} + \overbrace{\text{const}}^{C} \Big\} $$

Clearly, $C = 0$. But I'm not sure how to handle $A$ and $B$, particularly since $\Sigma = W W^{\top} + \Psi$.

*Derivative w.r.t. $\boldsymbol{\mu}$

The negative log-likelihood is:

$$ \mathcal{L} = \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) + \frac{n}{2} \ln |\Sigma| + \text{const} $$

The derivative of the two rightmost terms with respect to $\boldsymbol{\mu}$ is $0$, meaning we just need to compute:

$$ \frac{\partial}{\partial \boldsymbol{\mu}} \Big\{ \frac{1}{2} \sum_{i=1}^{n}(\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0 $$

By the linearity of differentiation, we have:

$$ \frac{1}{2} \sum_{i=1}^{n} \frac{\partial}{\partial \boldsymbol{\mu}} \Big\{ (\textbf{x}_i - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0 $$

Using Equation ($86$) from the Matrix Cookbox, we get:

$$ \frac{1}{2} \sum_{i=1}^{n} \Big\{ -2 \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} = 0 $$

Finally, solve for $\boldsymbol{\mu}$, we get:

$$ \begin{align} 0 &= \frac{1}{2} \sum_{i=1}^{n} \Big\{ -2 \Sigma^{-1} (\textbf{x}_i - \boldsymbol{\mu}) \Big\} \\ &= - \sum_{i=1}^{n} \Big\{ \Sigma^{-1} \textbf{x}_i - \Sigma^{-1} \boldsymbol{\mu} \Big\} \\ &= - \sum_{i=1}^{n} \Big\{ \Sigma^{-1} \textbf{x}_i \Big\} + n \Sigma^{-1} \boldsymbol{\mu} \\ - n \Sigma^{-1} \boldsymbol{\mu} &= - \Sigma^{-1} \sum_{i=1}^{n} \textbf{x}_i \\ \boldsymbol{\mu} &= \frac{1}{n} \sum_{i=1}^{n} \textbf{x}_i \end{align} $$

And we're done.


All those Greek letters are a pain to type, so let's use these variables $$\eqalign{ S = \Sigma,\,\,\,P = \Phi,\,\,\,L={\mathcal L},\,\,\,Z = (X-\mu 1) \cr }$$ where $X$ is the matrix whose columns are the $x_i$ vectors, and $(\mu 1)$ is a matrix all of whose elements are equal to $\mu$.

Further, let's use a colon to denote the trace/Frobenius product $$A:B = {\rm tr}(A^TB)$$ Write the objective function in terms of the Frobenius product and these new variables. Then find its differential and gradients. $$\eqalign{ L &= \tfrac{n}{2}\log(\det(S)) + \tfrac{1}{2}ZZ^T:S^{-1} + K \cr dL &= \tfrac{n}{2}{\rm tr\,}(d\log(S)) + \tfrac{1}{2}ZZ^T:dS^{-1} + 0 \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):dS \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):d(WW^T+P) \cr &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):(dW\,W^T+ W\,dW^T+dP) \cr }$$ Setting $dW=0$ yields the gradient wrt $P$ $$\eqalign{ dL &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):dP \cr \frac{\partial L}{\partial P} &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)\cr }$$ While setting $dP=0$ recovers the gradient wrt $W$ $$\eqalign{ dL &= \frac{1}{2}\Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big):(dW\,W^T+ W\,dW^T) \cr &= \Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)W:dW \cr \frac{\partial L}{\partial W} &= \Big(nS^{-1} - S^{-1}ZZ^TS^{-1}\Big)W \cr }$$ In several of the steps, we've made use of the fact that $S$ is symmetric.

  • $\begingroup$ Thanks! I can follow your derivation of $B$, the log of the determinant. I'm confused about how you setup for $A$ though. My part $A$ is $\frac{1}{2} \sum z^{\top} S z$. If I use the trace trick, that's: $\frac{1}{2} \sum \text{tr}(z z^{\top} S)$. But your setup is $\frac{1}{2} \text{tr}(z^{\top} z S^{-1})$. What happened to the summation? $\endgroup$ – gwg Jul 30 '18 at 19:47
  • $\begingroup$ Sorry, I misread the A term. The answer has been updated with the correct term. The change consisted of changing the $z$ vector into the $Z$ matrix. $\endgroup$ – greg Jul 30 '18 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.