41
$\begingroup$

I'm studying about EM-algorithm and on one point in my reference the author is taking a derivative of a function with respect to a matrix. Could someone explain how does one take the derivative of a function with respect to a matrix...I don't understand the idea. For example, lets say we have a multidimensional Gaussian function:

$$f(\textbf{x}, \Sigma, \boldsymbol \mu) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}}\exp\left( -\frac{1}{2}(\textbf{x}-\boldsymbol \mu)^T\Sigma^{-1}(\textbf{x}-\boldsymbol \mu)\right),$$

where $\textbf{x} = (x_1, ..., x_n)$, $\;\;x_i \in \mathbb R$, $\;\;\boldsymbol \mu = (\mu_1, ..., \mu_n)$, $\;\;\mu_i \in \mathbb R$ and $\Sigma$ is the $n\times n$ covariance matrix.

How would one calculate $\displaystyle \frac{\partial f}{\partial \Sigma}$? What about $\displaystyle \frac{\partial f}{\partial \boldsymbol \mu}$ or $\displaystyle \frac{\partial f}{\partial \textbf{x}}$ (Aren't these two actually just special cases of the first one)?

Thnx for any help. If you're wondering where I got this question in my mind, I got it from reading this reference: (page 14)

http://ptgmedia.pearsoncmg.com/images/0131478249/samplechapter/0131478249_ch03.pdf

UPDATE:

I added the particular part from my reference here if someone is interested :) I highlighted the parts where I got confused, namely the part where the author takes the derivative with respect to a matrix (the sigma in the picture is also a covariance matrix. The author is estimating the optimal parameters for Gaussian mixture model, by using the EM-algorithm):

$Q(\theta|\theta_n)\equiv E_Z\{\log p(Z,X|\theta)|X,\theta_n\}$

enter image description here

$\endgroup$
4
  • 1
    $\begingroup$ Possibly helpful: math.stackexchange.com/questions/94562/matrix-vector-derivative $\endgroup$
    – dreamer
    Dec 30, 2013 at 12:18
  • 1
    $\begingroup$ How is the function $Q(\theta|\theta_n)$ defined in the screenshot? $\endgroup$
    – dreamer
    Dec 30, 2013 at 12:24
  • 1
    $\begingroup$ @Dreamer I'll add it into the post. One sec, you can also see it from the reference pages 7-8 $\endgroup$
    – jjepsuomi
    Dec 30, 2013 at 12:25
  • 1
    $\begingroup$ I added the definition for $Q(\theta|\theta_n)$. However, I only require to understand how the author is doing the calculations in the reference. A simple example using multidimensional Gaussian is enough :) I can calculate it then myself for my specific problem (EM-algorithm). $\endgroup$
    – jjepsuomi
    Dec 30, 2013 at 12:29

2 Answers 2

36
$\begingroup$

It's not the derivative with respect to a matrix really. It's the derivative of $f$ with respect to each element of a matrix and the result is a matrix.

Although the calculations are different, it is the same idea as a Jacobian matrix. Each entry is a derivative with respect to a different variable.

Same goes with $\frac{\partial f}{\partial \mu}$, it is a vector made of derivatives with respect to each element in $\mu$.

You could think of them as $$\bigg[\frac{\partial f}{\partial \Sigma}\bigg]_{i,j} = \frac{\partial f}{\partial \sigma^2_{i,j}} \qquad \text{and}\qquad \bigg[\frac{\partial f}{\partial \mu}\bigg]_i = \frac{\partial f}{\partial \mu_i}$$ where $\sigma^2_{i,j}$ is the $(i,j)$th covariance in $\Sigma$ and $\mu_i$ is the $i$th element of the mean vector $\mu$.

$\endgroup$
3
  • 2
    $\begingroup$ +1 Ahaa :) that solves it then. Thank you, nothing further ;) $\endgroup$
    – jjepsuomi
    Dec 30, 2013 at 12:24
  • 3
    $\begingroup$ What you're saying is right, however, this doesn't quite explain how to do the calulations (it's not so easy to do this elementwise). $\endgroup$
    – dreamer
    Dec 30, 2013 at 12:25
  • $\begingroup$ @Dreamer It'd be glad to see some explanations on the calculations if someone wants to show them :) $\endgroup$
    – jjepsuomi
    Dec 30, 2013 at 12:28
10
$\begingroup$

You can view this in the same way you would view a function of any vector. A matrix is just a vector in a normed space where the norm can be represented in any number of ways. One possible norm would be the root-mean-square of the coefficients; another would be the sum of the absolute values of the matrix coefficients. Another is as the norm of the matrix as a linear operator on a vector space with its own norm.

What is significant is that the invertible matrices are an open set; so a derivative can make sense. What you have to do is find a way to approximate $$ f(x,\Sigma + \Delta\Sigma,\mu)-f(x,\Sigma,\mu)$$ as a linear function of $\Delta\Sigma$. I would use a power series to find a linear approximation. For example, $$ (\Sigma+\Delta\Sigma)^{-1}=\Sigma^{-1}(I+(\Delta\Sigma) \Sigma^{-1})^{-1} =\Sigma^{-1} \sum_{n=0}^{\infty}(-1)^{n}\{ (\Delta\Sigma)\Sigma^{-1}\}^{n} \approx \Sigma^{-1}(I-(\Delta\Sigma)\Sigma^{-1})$$ Such a series converges for $\|\Delta\Sigma\|$ small enough (using whatever norm you choose.) And, in the language of derivatives, $$ (\frac{d}{d\Sigma} \Sigma^{-1})\Delta\Sigma = -\Sigma^{-1}(\Delta\Sigma)\Sigma^{-1} $$ Remember, that the derivative is a linear operator on $\Delta\Sigma$; if you squint you can almost see the classical term $\frac{d}{dx}x^{-1} =-x^{-2}$. Chain rules for derivatives apply. So that's how you can handle the exponential composed with matrix inversion.

$\endgroup$
2
  • $\begingroup$ "using whatever norm you choose." - Is this because all norms are equivalent in $\mathbb{R}^n$ and the space of matrices is "kind of" one of these spaces? $\endgroup$
    – soap
    Apr 4, 2017 at 9:43
  • 1
    $\begingroup$ @Simoes Yes, all norms on the same finite-dimensional linear space are equivalent. $\endgroup$ Apr 4, 2017 at 14:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .