I've been studying the Expectation Maximization algorithm. According to the formula shown here, what I have to do in the M step is to compute a new $\theta$ that maximizes the conditional expectation of the log function, which is $\ln P[X, z|\theta] $:

Formally we have, $$ \begin{aligned} \theta_{n+1} &=\arg \max_{\theta}\left\{l\left(\theta \mid \theta_{n}\right)\right\} \\ &=\arg \max _{\theta}\left\{L\left(\theta_{n}\right)+\sum_{\mathbf{z}} \mathcal{P}\left(\mathbf{z} \mid \mathbf{X}, \theta_{n}\right) \ln \frac{\mathcal{P}(\mathbf{X} \mid \mathbf{z}, \theta) \mathcal{P}(\mathbf{z} \mid \theta)}{\mathcal{P}\left(\mathbf{X} \mid \theta_{n}\right) \mathcal{P}\left(\mathbf{z} \mid \mathbf{X}, \theta_{n}\right)}\right\} \end{aligned} $$ Now drop terms which are constant w.r.t. $\theta$ $$ \begin{aligned}{l} =\arg \max _{\theta}\left\{\sum_{\mathbf{z}} \mathcal{P}\left(\mathbf{z} \mid \mathbf{X}, \theta_{n}\right) \ln \mathcal{P}(\mathbf{X} \mid \mathbf{z}, \theta) \mathcal{P}(\mathbf{z} \mid \theta)\right\} \\ =\arg \max _{\theta}\left\{\sum_{\mathbf{z}} \mathcal{P}\left(\mathbf{z} \mid \mathbf{X}, \theta_{n}\right) \ln \frac{\mathcal{P}(\mathbf{X}, \mathbf{z}, \theta)}{\mathcal{P}(\mathbf{z}, \theta)} \frac{\mathcal{P}(\mathbf{z}, \theta)}{\mathcal{P}(\theta)}\right\} \\ =\underset{\theta}{\arg \max }\left\{\sum_{\mathbf{z}} \mathcal{P}\left(\mathbf{z} \mid \mathbf{X}, \theta_{n}\right) \ln \mathcal{P}(\mathbf{X}, \mathbf{z} \mid \theta)\right\} \\ =\arg \max _{\theta}\left\{\mathbf{E}_{\mathbf{Z} \mid \mathbf{X}, \theta_{n}}\{\ln \mathcal{P}(\mathbf{X}, \mathbf{z} \mid \theta)\}\right\} \end{aligned} $$ In Equation $(17)$ the expectation and maximization steps are apparent. The EM algorithm thus consists of iterating the:

  1. E-step: Determine the conditional expectation $\mathrm{E}_{\mathbf{Z} \mid \mathbf{X}, \theta_{n}}\{\ln \mathcal{P}(\mathbf{X}, \mathbf{z} \mid \theta)\}$
  2. M-step: Maximize this expression with respect to $\theta$.

( The excerpt above can be acquired in page 8 of this tutorial: http://www.seanborman.com/publications/EM_algorithm.pdf )

However, in the coin toss example below: http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html?pagewanted=all

$\ln P[X, z|\theta] $ is nowhere to be found, and they don't prove how the new $\theta^{t+1}$ they got after each iteration is better than the $\theta^t$ previously acquired.


1 Answer 1


Note that $$\sum_z P[z|X,\theta_n] \ln P[X,z|\theta] \\=\sum_\color{red}z\bigg(\ln P[X,z|\theta]\bigg)\color{red}{P[z|X,\theta_n]} \\=E_{Z|X,\theta_n}\bigg(\ln P[X,z|\theta]\bigg)$$ Here probability distribution is $P[z|X,\theta_n]$ and summing over $z$. So, by the definition of expectation we get the desired result.

  • $\begingroup$ I understand how the formulas work. However, what I really don't get is the way Theta is found in the coin example. They don't show or prove that the new found Theta will maximize the conditional expectation of the log function. $\endgroup$
    – IcySnow
    Commented May 18, 2013 at 5:52
  • 1
    $\begingroup$ @IcySnow: I think you need to edit your question and explain your problem more briefly. I don't understand your problem still and think others also not understand otherwise they can answer your question. $\endgroup$
    – Argha
    Commented May 18, 2013 at 6:04
  • $\begingroup$ I edited my question. Hopefully it makes more sense now. If there's still something wrong with it, please let me know and I will try my best to elaborate. $\endgroup$
    – IcySnow
    Commented May 18, 2013 at 6:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .