1
$\begingroup$

Consider now n i.i.d. observations of the vector data $({x_1,...,x_n}).$ Using the pdf, we can write the log-likelihood expression:

$$l(\boldsymbol x)=\sum_{i=1}^nln(\sum_{k=1}^K\pi_kp(x_i|\mu_k))$$

In order to estimate the unknown parameters $({π_k,μ_k,Σ_k})$ we must trace each data sample $x_j$ as to which of the k mixtures it belongs to. If an oracle would tell us, our task would be very easy; but we do not know it. This can be handled by using a new K-dimensional binary random vector $z∈(0,1)$ such that only one component at any one time is 1, all the rest being zero, and so $\sum_{i=1}^Kz_k=1$. In fact, also $p(z_k=1)=\pi_k.$ Notice that z is not observable, hence a hidden variable.

Thus for example,

$$p(x|z_k)=\prod_{j=1}^d\mu_{kj}^{xj}(1-\mu_{kj})^{1-xj}$$ $$p(x|z,\mu)=\prod_{i=1}^Kp(x|\mu_i)^{zi}\phantom{2}and\phantom{2}p(z|\pi)=\prod_{j=1}^K\pi_j^{zj}$$

How can we marginalize the product of p(x|z,μ) and p(z|π) (i.e., integrate out z) and obtain the below result:

$$p(x|\mu,\pi)=\sum_{k=1}^K\pi_kp(x|\mu_k)$$

$\endgroup$
1
$\begingroup$

Here's an explanation from a course I recently took an edX.org. It explains "Marginalizing over unobserved coordinates" for mixture models.

https://courses.edx.org/courses/course-v1:MITx+6.86x+1T2020/courseware/unit_4/P4_netflix/?child=first

It is on the 6th tab. If you can't get it, I just put an image here: enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.