# how does expectation maximization work?

I'm reading a tutorial on expectation maximization which gives an example of a coin flipping experiment (the description is at http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html?pagewanted=all). Could you please help me understand where the probabilities in step 2 of the process (i.e. in the middle of part b in the below illustration) come from? Thank you.

1. EM starts with an initial guess of the parameters. 2. In the E-step, a probability distribution over possible completions is computed using the current parameters. The counts shown in the table are the expected numbers of heads and tails according to this distribution. 3. In the M-step, new parameters are determined using the current completions. 4. After several repetitions of the E-step and M-step, the algorithm converges.
-

These are the likelihoods of the corresponding set of $10$ coin tosses having been produced by the two coins (using the current estimate for their biases) normalized to add up to $1$. The estimated probability of $k$ out of $10$ tosses of coin $i$ ($i\in\{A,B\}$) yielding heads is

$$p_i(k)=\left({10\atop k}\right) \theta_i^k (1-\theta_i)^{10-k}\;.$$

The binomial coefficient is the same for both coins, so it drops out in the normalization, and only the ratio of the remaining factors determines the result.

For instance, in the second row, we have $9$ heads and $1$ tails. Given the current bias estimates $\theta_A=0.6$ and $\theta_B=0.5$, the factors are

$$\theta_A^9 (1-\theta_A)^{10-9}\simeq0.004$$

and

$$\theta_B^9 (1-\theta_B)^{10-9}\simeq0.001\;,$$

resulting in the numbers

$$\frac{0.004}{0.004+0.001}=0.8$$

and

$$\frac{0.001}{0.004+0.001}=0.2$$

in the second row.

-
Thank you. So for each of 5 series of observations, we calculate the probability that the series is done by each coin by using the current bias estimates. Could you please help me understand: (a) how do we go from that probability distribution (0.8, 0.2) to the expectation (7.2H, 0.8T). I thought the expectation calculation cannot be (0.8*9 H, 0.8*1 T) because this way, we toss 10 times and only got 7.2+0.8 = 8 results? And (b) why do we need to do this? Many thanks again. – Martin08 Mar 18 '11 at 5:08
In case you haven't seen it: there's a follow-up question to your answer here. – t.b. Nov 11 '11 at 3:11
@Martin: Hi Martin, sorry I appear to have forgotten to answer your additional question -- I hope I've answered it now in my answer to the question t.b. links to above (in case it's still of any relevance to you...) – joriki Nov 11 '11 at 12:17
@joriki, have you implement this algorithm ? Because I implement it but found the result is A:0.66, B:0.66, which is different from the paper – zjffdu Nov 9 '13 at 11:21
@joriki can you see my answer. Is my understanding correct? – user13107 Jul 9 '14 at 6:57

Consider one of the coin-toss realizations in the figure.

Let $P(H_9T_1|A)$ be the probability of observing 9 heads, 1 tail when coin is A.

Let $P(H_9T_1|B)$ be the probability of observing 9 heads, 1 tail when coin is B.

Let $P(A|H_9T_1)$ be the probability of the coin being A when you observe 9 heads, 1 tail.

Let $P(B|H_9T_1)$ be the probability of the coin being B when you observe 9 heads, 1 tail.

Apply conditional probability definition.

$P(A|H_9T_1) = \frac{P(A) \cdot P(H_9T_1|A)}{P(H_9T_1)}$

$P(B|H_9T_1) = \frac{P(B) \cdot P(H_9T_1|B)}{P(H_9T_1)}$

Now,

$P(A) = 0.5 = P(B)$

Estimates of $P(H_9T_1|A)$ and $P(H_9T_1|B)$ are computed using method described by @joriki

Since the coin can either be A or B, $P(A|H_9T_1) + P(B|H_9T_1) = 1$

Hence you can calculate numbers in step 2. They are $P(A|H_9T_1)$ and $P(B|H_9T_1)$ respectively.

-
What is P(H9T1) ? Is it probability of getting 9 heads and 1 tail in unbiased coin? – Siddhesh Mar 15 at 4:08
• Let o1 <5H,5T>,o2 <9H,1T>,o3 <8H,2T>,o4 <4H,6T>,o5 <7H,3T> be five observation respectively.
• So each of these observation can either come from coin A with probability 0.5 or from coin B with probability 0.5
• If we select coin A , pA_Head denote the probability of Head from coin A , which is initialized as 0.60
• If we select coin B , pB_Head denote the probability of Head from coin B , which is initialized as 0.50

Now we have to find out weather these five observation has come from Coin A or Coin B. i.e we have to find P(A | o2) and P (B | o2) Now, P(A|o1) = P(A,o2) / P(o2)

• o2 can come from either Coin A or Coin B, So P(o2) = P(o2,A) + P(o2,B) [ Sum rule of Probability]
• P(A,o2) = P(A) * P(o2|A)
• P(B,o2) = P(B) * P(o2|B)