I'm reading through this paper on the expectation maximization algorithm and I'm trying to understand the part where the author introduces the incomplete data case.

We started with 2 coins of unknown bias $\theta_A$ and $\theta_B$, and we are given 2 vectors, $x = (x_0, x_1 \dots )$ where $0 \le x_i \le 10$ is the number of heads returned from ten flips on iteration $i$. For the complete data case we also have $z = (z_0, z_1, \dots)$ where $z_i \in [A, B]$ tells whether coin $A$ or $B$ was used. We don't have $z$ in the incomplete case, but before the author goes into an explanation of EM, they say the following:

One iterative scheme for obtaining completions could work as foloows: starting from some inital parameters, $\hat\theta^{(t)}=(\hat\theta_A^{(t)}, \hat\theta_B^{(t)})$, determine for each of the five sets whether coin A or coin B was more likely to have generated the observed flips (using the current parameter estimates). Then, assume these completions (that is, guessed coin assignments) to be correct, and apply the regular maximum likelihood estimation procedure to get $\hat\theta^{(t+1)}$. Finally, repeat these two steps until convergence.

I realize this is a stepping stone to the final solution but I don't understand how it could work. If you take the higher probability (from your initial guesses) and assume that coin was used on a set, and build $z$ in this way, iteration doesn't do anything. Once you have your initial $z$ and you re-estimate $\hat\theta_A$ and $\hat\theta_B$, subsequent iterations won't get you any closer to the real solution because you already maximized the likelihood for the biases.

In order for the $\hat\theta$ parameters to continue to move at all, they would need to change one or more $z_i$ but that will never happen. They provide an example in the illustration.

enter image description here

Now I don't know what the initial guess was here but if we carry out another iteration we get the following probabilities using these new estimates of the biases (rounded down).

set # | P(A) | P(B) |
    0 | 0.02 | 0.23 |
    1 | 0.26 | 0.00 |
    2 | 0.30 | 0.02 |
    3 | 0.00 | 0.23 |
    4 | 0.20 | 0.07 |

which will give a $z = (B, A, A, B, A)$ which will result in the same calculation for the estimates as before. This would be the case with any data set. So I'm not sure how the estimate continues to improve after the first iteration. Even with really bad initial guesses, after one shot aren't we stationary? I realize this isn't the focus of the paper but I don't understand it.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.