Our data can be represented as $X_{n\times d}$ matrix, where we have $n$ data points lying in $\mathbb{R}^d$. We assume that there are $k$ underlying Gaussian models, $\mathcal{N}_d(\mu_j, \Sigma_j)$, from which we could have drawn these points. In the Expectation-Maximization algorithm for finding these parameters (the probability of belonging to a certain distribution and the distribution parameters), we have four main calculations at every iteration:

  1. $W_{n\times k}$ - belief that $x_i\in X$ belongs to distribution $k$
  2. $\Pi_{k\times 1}$ - probability distribution of drawing from one of the $k$ models
  3. $M_{k\times d}$ - means of the $k$ distributions
  4. $\Sigma_{k\times d\times d}$ - covariance matrices ($d\times d$) for each of the $k$ distributions

I'm wondering if there are closed-form solutions for all four of the calculations. Two of them seem evident:

  • $\Pi = (W^\tau \vec{1})/n$
  • $M = ((W^\tau X)\odot\Pi^{-1})/n$

but the others are calculated by iterating over values:

  • $W_{ij} = \frac{\pi_jf(x_i;\mu_j,\Sigma_j)}{\sum_{\ell=1}^{k}\pi_\ell f(x_i;\mu_\ell,\Sigma_\ell)}$, where $f$ is the probability density function
  • $\Sigma_j' = \frac{1}{n\pi_j'}\sum_{i=1}^{n}W_{ij}(x_i-\mu_j')^\tau(x_i-\mu_j')$

Where the $'$ indicates we found that value during the current iteration. Is there a different way to store these to get closed-form solutions? Or should I just give up and just sum them out?

  • $\begingroup$ Did you ever figure out the answer to this? I'd be curious to learn more. $\endgroup$ – Adam_G Nov 9 '17 at 23:48
  • $\begingroup$ I didn't, unfortunately. I think I ended up just writing the iterative implementations described above. $\endgroup$ – Brad Flynn Nov 10 '17 at 0:05
  • $\begingroup$ Oh well. That makes sense. Thank you $\endgroup$ – Adam_G Nov 10 '17 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.