2
$\begingroup$

I am trying to peace tighter the EM algorithm but I am not able to follow certain steps:

What I know:

Marginal pdf: $$f_{X_{1}}\left(x_{1}\right)=\int_{-\infty}^{\infty} f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right) d x_{2}$$

for all $x_1$ element of domain of $X_1$

Conditional Distributions:

$$p_{X_{2} \mid X_{1}}\left(x_{2} \mid x_{1}\right)=\frac{p_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right)}{p_{X_{1}}\left(x_{1}\right)}$$

MLE:

$$L(\theta ; \mathbf{x})=\prod_{i=1}^{n} f\left(x_{i} ; \theta\right), \quad \theta \in \Omega$$

$$l(\theta)=\log L(\theta)=\sum_{i=1}^{n} \log f\left(x_{i} ; \theta\right), \quad \theta \in \Omega$$

$$ k(\mathbf{z} \mid \theta, \mathbf{x})=\frac{h(\mathbf{x}, \mathbf{z} \mid \theta)}{g(\mathbf{x} \mid \theta)} (6.6.1) $$

What I am trying to follow:

The observed likelihood function is $L(\theta \mid \mathbf{x})=g(\mathbf{x} \mid \theta) .$ The complete likelihood function is defined by $$ L^{c}(\theta \mid \mathbf{x}, \mathbf{z})=h(\mathbf{x}, \mathbf{z} \mid \theta) $$ Our goal is maximize the likelihood function $L(\theta \mid \mathbf{x})$ by using the complete likelihood $L^{c}(\theta \mid \mathbf{x}, \mathbf{z})$ in this process. Using (6.6.1), we derive the following basic identity for an arbitrary but fixed $\theta_{0} \in \Omega:$ $$ \begin{aligned} \log L(\theta \mid \mathbf{x}) &=\int \log L(\theta \mid \mathbf{x}) k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\ &=\int \log g(\mathbf{x} \mid \theta) k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\ &=\int[\log h(\mathbf{x}, \mathbf{z} \mid \theta)-\log k(\mathbf{z} \mid \theta, \mathbf{x})] k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\ &=\int \log [h(\mathbf{x}, \mathbf{z} \mid \theta)] k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z}-\int \log [k(\mathbf{z} \mid \theta, \mathbf{x})] k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\ &=E_{\theta_{0}}\left[\log L^{c}(\theta \mid \mathbf{x}, \mathbf{Z}) \mid \theta_{0}, \mathbf{x}\right]-E_{\theta_{0}}\left[\log k(\mathbf{Z} \mid \theta, \mathbf{x}) \mid \theta_{0}, \mathbf{x}\right] \end{aligned} $$ where the expectations are taken under the conditional pdf $k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right)$. Define the first term on the right side of $(6.6 .3)$ to be the function $$ Q\left(\theta \mid \theta_{0}, \mathbf{x}\right)=E_{\theta_{0}}\left[\log L^{c}(\theta \mid \mathbf{x}, \mathbf{Z}) \mid \theta_{0}, \mathbf{x}\right] $$

Q1:

from the definition of observed likelihood and complete likelihood above I don't see how this $$\log L(\theta \mid \mathbf{x}) =\int \log L(\theta \mid \mathbf{x}) k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\$$

equations has come about.

Q2:

If I understand Q1 I might be able to understand why Log is only taken on $g(x|omega)$ And if this is correct, then I do given(6.6.1) understand the steps log(a/b)=log(a)-log(b) so I can follow down to line 4.

Q3: I don't understand what happening from line 4 to the last line, what do they mean by: where the expectations are taken under the conditional pdf $k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right)$.

and how k(...) term disappear and we get a capital Z and the conditional part of k is now the condition inside the Expectation.

$\endgroup$
4
  • $\begingroup$ You have some text repeated in your question. Could you please remove it ? Further, do you really need a new letter for everything ? I find it confusing instead of using just p, but maybe it is just me... $\endgroup$
    – Thomas
    Commented Jun 3, 2021 at 8:23
  • $\begingroup$ Out of interest, where is the presentation of EM that you are following ? $\endgroup$
    – Thomas
    Commented Jun 3, 2021 at 10:11
  • 1
    $\begingroup$ ( I removed the duplicated text by the way ) $\endgroup$
    – Thomas
    Commented Jun 3, 2021 at 10:15
  • 1
    $\begingroup$ @Thomas I am actually not sure I searched in google for explanation of EM algorithm a while a go, and then I had stored down a bunch of pdf PowerPoints. Just had a look at it again and got stuck, but the pdfs have references (1).(2) but the end of the slide is not provided so would expect the reference to the original text would be there.Might be that I just cut out what I was interested in at the time. can't remember. It might have been a snippet of a textbook when looking at it. Let me see if I can find it. $\endgroup$
    – ALEXANDER
    Commented Jun 3, 2021 at 10:19

1 Answer 1

0
$\begingroup$

Q1 :

$$\log L(\theta \mid \mathbf{x}) =\int \log L(\theta \mid \mathbf{x}) k\left(\mathbf{z} \mid \theta_{0}, \mathbf{x}\right) d \mathbf{z} \\$$

Just take $\log L(\theta \mid \mathbf{x})$ out of the integral (does not depend on $z$ and note that $k$ is a distribution over $z$, i.e. it is normalized. Essentially you are multiplying the left member by one.

Q3 :

Expressions like $\int f(x)p(x)dx$, are exactly the expected value of $f(X)$ when $X\sim p$, indicated sometimes as $E[f(X)]$ or also $E_{X\sim p}[f(X)]$. If $p$ is parametric, also one could use $E_{\theta}[f(X)]$ and the $p$ magically disappears. In your case $p(x)$ is the conditional distribution $k$ and you have more other variables but I thought this piece was the one confusing you.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .