1
$\begingroup$

For the expectation maximization of Gaussian mixture model with respect to $\Sigma_k$ we have:

$\frac{\partial L}{\partial \Sigma_k } =0$

$L = \sum_{i=1}^n \log(\sum_{k=1}^K \pi_K \mathcal{N}(x_i|\mu_k,\Sigma_k))$

$ \sum_{i=1}^N\frac{ \pi_K }{\sum_{k=1}^K \pi_K \mathcal{N}(x_i|\mu_k,\Sigma_k)} \frac{\partial}{\Sigma_k}(\frac{1}{\sqrt{(2\pi)^t|\Sigma_k|}}e^{-\frac{1}{2}(x_i-\mu_k)^T\Sigma_k^{-1}(x_i-\mu_k)}) = 0$

The answer is

$\Sigma_k = \sum_{n=1}^N\frac{1}{N_k}\frac{\pi_k\mathcal{N}(x_n|\mu_k,\Sigma_k)}{\Sigma_{j=1}^{K}\mathcal{\pi_j\mathcal{N}(x_n|\mu_j,\Sigma_j)}}(x_n - \mu_k)(x_n-\mu_k)^T$

$N_k = \sum_{n=1}^{N} \frac{\pi_k\mathcal{N}(x_n|\mu_k,\Sigma_k)}{\Sigma_{j=1}^{K}\mathcal{\pi_j\mathcal{N}(x_n|\mu_j,\Sigma_j)}}$

I cannot find the answer above. I assumed $|\Sigma_k|$ is determinant of $\Sigma_k$ . Is that correct? how should we perform the derivative with respect to matrix $\Sigma_k$ when one of place it is represented as determinant and in one place as inverse matrix.

$\endgroup$
2
  • $\begingroup$ @reuns Thanks for your comment. Yes thats right, but I wanted to find how the derivative is calculated exactly. For example how can we find the derivative of $(x_i-\mu_k)^T\Sigma_k^{-1}(x_i-\mu_k)$ with respect to $\Sigma_k$? $\endgroup$
    – Crimson
    Commented Jul 22, 2017 at 1:23
  • $\begingroup$ It works better now :) $\endgroup$
    – reuns
    Commented Jul 22, 2017 at 2:03

1 Answer 1

2
$\begingroup$

In the EM algorithm we take some $\log$ probabilities so that the maximization step for a Gaussian mixture is a quadratic optimization problem with a closed-form.


Let $S = \Sigma_k^{-1}, y_n = x_n-\mu_k,a_n = \frac{\mathbb{P}(x_n \in C_k)}{\sum_m \mathbb{P}(x_m \in C_k)}$. Then you want find $S$ minimizing $$J(S) = -\log \det|S|+\sum_n a_n y_n^T S y_n, \qquad (a_n > 0, y_n \in \mathbb{R}^m,S \in \mathbb{R}^{m \times m})$$ Due to the adjugate matrix formula $\det(S) I = S\, Adj(S)$ you'll find $$\frac{\partial }{\partial S_{i,j}} \log \det(S) = \frac{Adj(S)_{i,j}}{\det S}, \qquad \frac{\partial }{\partial S_{i,j}} J(S) = -\frac{Adj(S)}{\det S}+\sum_n a_n y_n(i) y_n(j), \qquad \frac{\partial }{\partial S} J(S)= -\frac{Adj(S)}{\det S}+\sum_n a_n y_n y_n^T$$ $$\frac{\partial }{\partial S} J(S)=0 \qquad \implies \qquad \frac{Adj(S)}{\det(S)} =S^{-1}= \sum_n a_n y_n y_n^T$$

And you'll need to compute $ \Sigma_k^{-1} = S$ as $\Sigma_k^{-1}$ is what we need to compute the $\log$ probabilities for the next step.

$\endgroup$
5
  • $\begingroup$ Thank you for the answer. how did you wrote J(S)? Specifically what is $a_n$ and $P$? Is there any reference that I can study to understand this? If we choose $S$ as inverse of $\Sigma$ then shouldn't we find the derivative with respect to $S^{-1}$ since the derivative is with respect to $\Sigma$? $\endgroup$
    – Crimson
    Commented Jul 22, 2017 at 1:42
  • $\begingroup$ @Crimson Everything is explained on wikipedia or any course on the EM algorithm. You should start with the equations for the $2$ Gaussian mixture case (as on wikipedia). Also take a look at the alternating least square, a general way to find approximate solution for $\min f(X,Y)$ where $f$ is quadratic or linear in $X$ and $Y$ alone but not in both (example : the SVD). $\endgroup$
    – reuns
    Commented Jul 22, 2017 at 1:44
  • $\begingroup$ Thanks for your detailed explanation. I still don't understand some part od the answer. We assumed $S=\Sigma_k^{-1}$ shouldn't we take the derivative with respect to $\Sigma_k$ (or $S^{-1}$)? but based on my understanding you wrote the derivative with respect to S. I also don't understand what is $a_n = \mathbb{P}(x_n|x_n \in C_k)$. I would appreciate further insights on this. $\endgroup$
    – Crimson
    Commented Jul 22, 2017 at 2:13
  • $\begingroup$ @Crimson With $S = \Sigma_k^{-1}$ we want to minimize over $S$ the function $J(S)$ I wrote. $a_n$ are the normalized probabilities.. Take a look at the logarithm of the multivariate Gaussian $\endgroup$
    – reuns
    Commented Jul 22, 2017 at 2:58
  • $\begingroup$ Thanks for the explanation and link. I will look at it. $\endgroup$
    – Crimson
    Commented Jul 22, 2017 at 3:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .