0
$\begingroup$

In chapter 9 of Pattern Recognition And Machine Learning book an alternative version of Expectation Maximization Algorithm(EM) is introduced as follows:

$X$: observed data

$Z$: all latent variables

$\theta$: is a set of all model paramters

The log likelihood function is shown as: $\ln p(X|\theta) = \ln\{ \sum_Z p(X,Z|\theta)\}$

We are not given the complete data set $\{X,Z\}$, but only the incomplete data $X$. The value of latent variables in $Z$ is given by $p(Z|X,\theta)$. Because we cannot use the complete-data log likelihood, we consider instead its expected value under the posterior distribution of the latent variable, which corresponds to the E step of EM algorithm. In subsequent M step, we maximize the expectation. If current estimate for the parameter is denoted as $\theta ^ {old}$, then the pair of successive E and M step give rise to $\theta ^ {new}$.

In the E step, we use $\theta ^{old}$ to find posterior distribution of $Z$ by $p(Z|X,\theta^{old})$. We then use this posterior distribution to find the expectation of the complete-data log likelihood evaluated for some general parameter value $\theta$. This expectation, denoted $\mathcal{Q}(\theta, \theta ^{old})$, is given by

$\mathcal{Q}(\theta, \theta ^{old}) = \sum_z p(Z|X,\theta ^{old}) \ln{p (X,Z|\theta)} \quad\quad\quad (9.30)$

In the M step we determined revised parameter estimate $\theta ^{new}$ by maximizing function:

$\theta ^{new} = \text{arg}\max \limits_{\theta} \mathcal{Q}(\theta,\theta^{old}) \quad\quad\quad (9.31)$

I cannot follow how 9.30 is written.

Based on which fact $\mathcal{Q}(\theta, \theta ^{old})$ is written as it is shown in 9.30? Why in the equation 9.30, it used $\ln{p (X,Z|\theta)}$ instead of $p (X,Z|\theta)$?

Could anyone help to understand this please? Thanks in advance.

$\endgroup$
1
$\begingroup$

$\mathcal{Q}(\theta, \theta^{old})$ is defined to be the expectation of the complete data log likelihood evaluated for some general parameter value $\theta$.

\begin{align}\mathcal{Q}(\theta , \theta^{old})&=\mathbb{E}_{Z|X,\theta^{old}} \ln p(X,Z|\theta)\\ &=\sum_Z p(Z|X, \theta^{old})\ln p(X,Z|\theta)\end{align}

$\endgroup$
6
  • $\begingroup$ Thanks for your answer. I just know the expected value of $X$ is calculated as $E(x) = \sum_{i=1}^{\infty} x_i p_i$ and $ \ln p(x|\theta) = \ln \{ \sum_z p(X,Z|\theta) \}$(assuming $\ln$ is used for the purpose of maximization using derivative). But I still don't understand the initial idea for writing the $\mathcal{Q}(\theta, \theta ^{old}) $. Would you please explain more. Thanks $\endgroup$ – Crimson Jul 21 '17 at 15:04
  • 1
    $\begingroup$ $Z$ is a latent variable here, they are the missing values. Using $\theta^{old}$, we can compute $p(Z|X,\theta^{old})$, and hence perform the estimation $\mathbb{E}_{Z|X,\theta^{old}}$. It's like there is a missing value here, let's estimate it. Once we estimate a $Z$, we can evaluate $P(X,Z|\theta)$ and compute $Q(\theta, \theta^{old})$. Note that $\theta$ is a variable that is to be optimized in the $M$ step. $\endgroup$ – Siong Thye Goh Jul 21 '17 at 16:41
  • $\begingroup$ Thank you very much for your time. When we say $\mathcal{Q}(\theta,\theta ^{old})$ is expectation of complete data log likelihood means it can be written as $E(\ln p(X|\theta))$? If yes, how it is written as 9.30? Sorry if I am asking too many questions. $\endgroup$ – Crimson Jul 21 '17 at 16:53
  • $\begingroup$ complete data refers to $(X,Z)$. $X$ is observed but $Z$ is not observed, hence we say that $X$ is imcomplete. $\endgroup$ – Siong Thye Goh Jul 21 '17 at 16:54
  • 1
    $\begingroup$ they wrote $\mathbb{E} (\ln p(X,Z |\theta)$ which agree with your expresssion, note that expectation depends on the distribution that you consider, in this case the distribution is $P(Z|X, \theta^{\old})$. $\endgroup$ – Siong Thye Goh Jul 21 '17 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.