Bayesian Mixtures of Gaussians Notation

I am having trouble understanding the following probability notations: (sorry about the page break)

$$\bullet$$ Consider the Bayesian mixture of Gaussians,

1. Draw $$\mu _ { k } \sim \mathcal { N } \left( 0 , \tau ^ { 2 } \right)$$ for $$k = 1 \ldots K$$ .
2. For $$i = 1 \ldots n :$$

(a) Draw $$z _ { i } \sim \operatorname { Mult } ( \pi )$$

(b) Draw $$x _ { i } \sim \mathcal { N } \left( \mu _ { z _ { i } } , \sigma ^ { 2 } \right)$$ Can someone please tell me what does "draw $$z_i \sim Multi(π)$$" mean?

This is my closest guess: Choose k $$\mu$$s, from distribution $$\mathcal N(0,τ^2)$$. Then do the following $$n$$ times: - Randomly select a $$\mu$$ from the $$\mu$$s picked before, and randomly select an $$x$$ from the distribution $$\mathcal N(u,σ^2)$$.

By the way, this is from a tutorial on variational inference. See bottom of page 1.

Thanks!

Think about it like the following:

$$z_i \sim Mult(\pi)$$ where $$\pi$$ is a $$\textit{vector}$$ of probabilities of size $$K$$. The multinomial distribution is a generalization of the categorical distribution, which the poster below correctly mentions. The categorical distribution is equivalent to considering the multinomial when we have one trial and more than 2 categories for consideration. In other words, the author of the document could have replace the multinomial specification with the categorical and would have been more easily interpreted.

As a very concrete example, let's consider the case where you have just completed step (1) above and have drawn all the means, $$\mu_k$$ for $$k = 1, \ldots, K$$. Now imagine we have a $$K$$-sided die that you can roll, where each side has probability $$\pi_k$$ of being chosen for $$k = 1, \ldots, K$$. (Normally each side of a dice roll is equally likely but here they may not be equal). These $$\pi_k$$ can be collected into a vector $$\pi = (\pi_{1}, \ldots, \pi_K)$$, which is what we put into the "Mult" specification in part (a) above. Then, the "action":

$$z _ { i } \sim \operatorname { Mult } ( \pi )$$

is the same as rolling the $$K$$-sided die and setting $$z_i$$ equal to the result of of the dice roll. Here $$z_i$$ can take values in the range of $$\{1, \ldots, K\}$$. Then, $$z_i$$ will serve as the index of which of the means $$\mu_k$$ you had generated earlier will be used to now serve as the mean of the sampling distribution of $$x_i$$. Now repeat this process for each $$i \in \{1, \ldots, n\}$$.

Finally, let's consider a real-life example. Suppose $$K=3$$ and $$n=2$$, then we will first in step (1) draw out three means $$\mu_1, \mu_2, \mu_3$$. Then, we have a 3-sided die for which we may roll a $$1$$ with probability $$\pi_1$$, a $$2$$ with probability $$\pi_2$$, and a $$3$$ with probability $$\pi_3$$. Then, $$\pi = (\pi_1, \pi_2, \pi_3)$$. Then we want to do parts (a) and (b) twice since we set $$n=2$$.

Roll the 3-sided die and record what we got. Let's say for the $$n=1$$ case we rolled a $$2$$. Then, we set $$z_1 = 2$$. For the $$n=2$$ case let's say we rolled a $$3$$. Then we set $$z_2 = 3$$. Now, we have:

Take the $$z_1$$ and $$z_2$$ values to serve as the index of which of the three means we want to use for sampling $$x_1$$ and $$x_2$$. This entails sampling:

$$x _ { 1 } \sim \mathcal { N } \left( \mu _ { 2 } , \sigma ^ { 2 } \right)$$

and

$$x _ { 2 } \sim \mathcal { N } \left( \mu _ { 3 } , \sigma ^ { 2 } \right)$$

and plugging in the generated $$\mu_2$$ and $$\mu_3$$ from part (1) above, since $$\mu_{z_1} = \mu_{2}$$ and $$\mu_{z_2} = \mu_{3}$$.

• There was a bug with stackexchange, I've reposted my answer above. – user321627 Feb 8 at 8:25
• very clear. thanks! – AlphaBeta Feb 8 at 17:14
• You're welcome, if you feel this has helped please upvote or select as the main answer! – user321627 Feb 9 at 5:48