# Expectation Maximization Algorithm with latent variable

In chapter 9 of Pattern Recognition And Machine Learning book an alternative version of Expectation Maximization Algorithm(EM) is introduced as follows:

$X$: observed data

$Z$: all latent variables

$\theta$: is a set of all model paramters

The log likelihood function is shown as: $\ln p(X|\theta) = \ln\{ \sum_Z p(X,Z|\theta)\}$

We are not given the complete data set $\{X,Z\}$, but only the incomplete data $X$. The value of latent variables in $Z$ is given by $p(Z|X,\theta)$. Because we cannot use the complete-data log likelihood, we consider instead its expected value under the posterior distribution of the latent variable, which corresponds to the E step of EM algorithm. In subsequent M step, we maximize the expectation. If current estimate for the parameter is denoted as $\theta ^ {old}$, then the pair of successive E and M step give rise to $\theta ^ {new}$.

In the E step, we use $\theta ^{old}$ to find posterior distribution of $Z$ by $p(Z|X,\theta^{old})$. We then use this posterior distribution to find the expectation of the complete-data log likelihood evaluated for some general parameter value $\theta$. This expectation, denoted $\mathcal{Q}(\theta, \theta ^{old})$, is given by

$\mathcal{Q}(\theta, \theta ^{old}) = \sum_z p(Z|X,\theta ^{old}) \ln{p (X,Z|\theta)} \quad\quad\quad (9.30)$

In the M step we determined revised parameter estimate $\theta ^{new}$ by maximizing function:

$\theta ^{new} = \text{arg}\max \limits_{\theta} \mathcal{Q}(\theta,\theta^{old}) \quad\quad\quad (9.31)$

I cannot follow how 9.30 is written.

Based on which fact $\mathcal{Q}(\theta, \theta ^{old})$ is written as it is shown in 9.30? Why in the equation 9.30, it used $\ln{p (X,Z|\theta)}$ instead of $p (X,Z|\theta)$?

$\mathcal{Q}(\theta, \theta^{old})$ is defined to be the expectation of the complete data log likelihood evaluated for some general parameter value $\theta$.
• Thanks for your answer. I just know the expected value of $X$ is calculated as $E(x) = \sum_{i=1}^{\infty} x_i p_i$ and $\ln p(x|\theta) = \ln \{ \sum_z p(X,Z|\theta) \}$(assuming $\ln$ is used for the purpose of maximization using derivative). But I still don't understand the initial idea for writing the $\mathcal{Q}(\theta, \theta ^{old})$. Would you please explain more. Thanks – Crimson Jul 21 '17 at 15:04
• $Z$ is a latent variable here, they are the missing values. Using $\theta^{old}$, we can compute $p(Z|X,\theta^{old})$, and hence perform the estimation $\mathbb{E}_{Z|X,\theta^{old}}$. It's like there is a missing value here, let's estimate it. Once we estimate a $Z$, we can evaluate $P(X,Z|\theta)$ and compute $Q(\theta, \theta^{old})$. Note that $\theta$ is a variable that is to be optimized in the $M$ step. – Siong Thye Goh Jul 21 '17 at 16:41
• Thank you very much for your time. When we say $\mathcal{Q}(\theta,\theta ^{old})$ is expectation of complete data log likelihood means it can be written as $E(\ln p(X|\theta))$? If yes, how it is written as 9.30? Sorry if I am asking too many questions. – Crimson Jul 21 '17 at 16:53
• complete data refers to $(X,Z)$. $X$ is observed but $Z$ is not observed, hence we say that $X$ is imcomplete. – Siong Thye Goh Jul 21 '17 at 16:54
• they wrote $\mathbb{E} (\ln p(X,Z |\theta)$ which agree with your expresssion, note that expectation depends on the distribution that you consider, in this case the distribution is $P(Z|X, \theta^{\old})$. – Siong Thye Goh Jul 21 '17 at 18:06