Derivative of negative log-likelihood function for data following multivariate Gaussian distribution

Question

I'm trying to find the gradient of the negative log-likelihood function for data following a multivariate Gaussian distribution. The negative log-likelihood function is given by $$L(\pmb{\theta}) =-\ln p(\mathcal{D}|\pmb{\theta}) = \frac{1}{2}\ln |\pmb{\Sigma}(\pmb{\theta})| + \frac{1}{2}(\pmb{y}-\pmb{\mu}(\pmb{\theta}))^{T}\pmb{\Sigma}^{-1}(\pmb{\theta})(\pmb{y}-\pmb{\mu}(\pmb{\theta})) $$ where $\pmb{y}$ and $\pmb{\mu}(\pmb{\theta})$ are $N\times1$ vectors and $\pmb{\Sigma}(\pmb{\theta})$ is the $N\times N$ covariance matrix. $\pmb{\theta}$ is the $n_{p}\times 1$ vector of parameters on which both the mean vector $\pmb{\mu}$ and the covariance matrix $\pmb{\Sigma}$ depend.

My objective is to differentiate $L(\pmb{\theta})$ with respect to $\pmb{\theta}$. I'll be using the denominator layout.

$$\frac{dL(\pmb{\theta})}{d\pmb{\theta}} = \frac{d\pmb{\Sigma}}{d\pmb{\theta}}\frac{\partial L(\pmb{\theta})}{\partial \pmb{\Sigma}} + \frac{d\pmb{\mu}}{d\pmb{\theta}}\frac{\partial L(\pmb{\theta})}{\partial \pmb{\mu}}. $$ Now, $\frac{dL(\pmb{\theta})}{d\pmb{\theta}}$ will be of size $n_{p}\times 1$. On the right hand side, $\frac{d\pmb{\mu}}{d\pmb{\theta}}$ and $\frac{\partial L(\pmb{\theta})}{\partial \pmb{\mu}}$ have sizes $n_{p}\times N$ and $N\times 1$ respectively. So, the second term has the same size as the left hand side. However, I'm unable to carry out the correct multiplication operation between the two derivatives in the first term on the right hand size.

Could anyone please help me figure out how to do this?

Answer 1

However, I'm unable to carry out the correct multiplication operation between the two derivatives in the first term on the right hand size.

This is because it is not a matrix multiplication. Note that by definition, the derivative of $f：U→V$ at a point $u$ is a linear function $𝐃f(x)：U→V$, and the chain rule states $𝐃[g∘f](x) = 𝐃g(f(x))∘𝐃f(x)$.

There is no multiplication here anywhere; it only enters the picture if we identify linear functions as tensors, and the composition of linear functions as tensor contractions.

In the case at hand, $\frac{𝖽 𝚺}{𝖽𝛉}$ is the derivative of a function $ℝ^{n_p} →ℝ^{N}⊗ℝ^{N}$, hence if would be identified with a 3-dimensional tensor $\frac{𝖽 𝚺}{𝖽𝛉} ∈ ℝ^{N}⊗ℝ^{N} ⊗ (ℝ^{n_p})^*$. You now have 3 options:

1. Deal with the higher order tensors and tensor-contractions
2. Avoid them by using back-propagation.
3. Avoid them by using vectorization (cf. Steph's answer)

Solution 1 - deal with the higher order tensors.

I'll be using the denominator layout.

Note that this is an insufficient specification when dealing with higher order tensors.

$$\begin{aligned} \underbrace{𝐜}_{1, n_p} = \underbrace{\tfrac{𝖽 L(𝛉)}{𝖽𝚺}}_{1,(N×N)} \underbrace{\tfrac{𝖽𝚺}{𝖽𝛉}}_{(N×N), n_p} &= \bigg(\sum_{ij} \Big(\tfrac{𝖽 L(𝛉)}{𝖽𝚺}\Big)_{ij} \Big(\tfrac{𝖽𝚺}{𝖽𝛉}\Big)_{ij, k}\bigg)_k \\&=\bigg(\sum_{ij} \Big(\tfrac{𝖽 L(𝛉)}{𝖽𝚺_{ij}}\Big) \Big(\tfrac{𝖽𝚺_{ij}}{𝖽𝛉_k}\Big)\bigg)_k =\Big(\big\langle\tfrac{𝖽 L(𝛉)}{𝖽𝚺}，\tfrac{𝖽𝚺}{𝖽𝛉_k}\big\rangle_F\Big)_k \end{aligned}$$

Solution 2 - Using backpropagation

We roll out the chain rule in functional form and then aggregate from the left.

$$\begin{aligned} \tfrac{𝖽 L(𝛉)}{𝖽𝛉} &= \underbrace{\tfrac{𝖽 L(𝛉)}{𝖽(𝛍, 𝚺)}}_{ℝ^N ⊕ (ℝ^N ⊗ ℝ^N)→ℝ} ∘ \underbrace{\tfrac{𝖽(𝛍, 𝚺)}{𝖽𝛉}}_{ℝ^{n_p} → ℝ^N ⊕ (ℝ^N ⊗ ℝ^N)} \\&= [(∆𝛍, ∆𝚺) → \tfrac{𝖽 L(𝛉)}{𝖽(𝛍, 𝚺)}(∆𝛍, ∆𝚺)] ∘ [∆𝛉 ↦ \tfrac{𝖽(𝛍, 𝚺)}{𝖽𝛉}(∆𝛉)] \\&= [∆𝛉 ↦ \tfrac{𝖽 L(𝛉)}{𝖽(𝛍, 𝚺)}\big(\tfrac{𝖽(𝛍, 𝚺)}{𝖽𝛉}(∆𝛉)\big)] \end{aligned}$$

---

In both cases you can apply the formulas

$$\begin{aligned} &(1)&&\frac{𝖽\log\det 𝚺}{𝖽 𝚺} \;\hat{=}\; 𝚺^{-⊤} \;\hat{=}\; [∆𝚺 ↦ ⟨𝚺^{-⊤}∣∆𝚺⟩_F] \\&(2)&&\frac{𝖽 𝐮^⊤𝚺^{-1}𝐮 }{𝖽 𝚺} \;\hat{=}\; -𝚺^{-⊤}𝐮𝐮^⊤𝚺^{-1} \;\hat{=}\; [∆𝚺 ↦ ⟨-𝚺^{-⊤}𝐮𝐮^⊤𝚺^{-1}∣∆𝚺⟩_F] \end{aligned}$$

Answer 2

Using differential, \begin{eqnarray} dL &=& \frac{\partial L}{\partial \mathrm{vec}(\mathbf{\Sigma})}:d\mathrm{vec}(\mathbf{\Sigma})+ \frac{\partial L}{\partial \pmb{\mu}}:d\pmb{\mu} \\ &=& \frac{\partial L}{\partial \mathrm{vec}(\mathbf{\Sigma})}:\mathbf{J}_\Sigma d\pmb{\theta}+ \frac{\partial L}{\partial \pmb{\mu}}:\mathbf{J}_\mu d\pmb{\theta} \end{eqnarray} So the gradient writes $$ \frac{\partial L}{\partial \pmb{\theta}} = \mathbf{J}_\Sigma^T \frac{\partial L}{\partial \mathrm{vec}(\mathbf{\Sigma})}+ \mathbf{J}_\mu^T \frac{\partial L}{\partial \pmb{\mu}} $$ Here the colon operator : denotes the Frobenius inner product (here the usual inner product between vectors). The matrices $\mathbf{J}$ are Jacobian matrices that depend on your parameterization. The $(i,j)$-th element of $\mathbf{J}_\mu$ is defined as $\frac{\partial \mu_i}{\partial \theta_j}$. Similar definition applies to the other Jacobian matrix.

Here $\mathrm{vec}$ is the vectorization operation that transforms a matrix into a vector (by stacking its columns)

Answer 3

The log-likelihood is: $$\ell=\log\mathcal L=-\frac12\left(\log\left|2\pi\Sigma\right|+\mathrm{tr}(\Sigma^{-1}(y-\mu)(y-\mu)^T\right)$$ where $|A|$ is the determinant of the matrix $A$ and $\mathrm{tr}(A)$ is the trace of matrix $A$. I used the cyclic property of the trace to rewrite it in this format.

The derivatives with regard to $\mu$ and $\Sigma$ are thus: $$\frac{d\ell}{d\mu}=\Sigma^{-1}(y-\mu)$$ $$\frac{d\ell}{d\Sigma}=\frac12\left(\Sigma^{-1}(y-\mu)(y-\mu)^T\Sigma^{-1} - \Sigma^{-1}\right)$$

You can use a symbolic calculator (like http://www.matrixcalculus.org) for example to perform the derivatives, if you want to avoid the pain.

To connect these back to $\theta$ then you probably need to multiply these results by the appropriate jacobian (in the right dimension), if necessary.