I am having trouble understanding the following probability notations: (sorry about the page break)

$\bullet$ Consider the Bayesian mixture of Gaussians,

  1. Draw $\mu _ { k } \sim \mathcal { N } \left( 0 , \tau ^ { 2 } \right)$ for $k = 1 \ldots K$ .
  2. For $i = 1 \ldots n :$

(a) Draw $z _ { i } \sim \operatorname { Mult } ( \pi )$

(b) Draw $x _ { i } \sim \mathcal { N } \left( \mu _ { z _ { i } } , \sigma ^ { 2 } \right)$ Can someone please tell me what does "draw $z_i \sim Multi(π)$" mean?

This is my closest guess: Choose k $\mu$s, from distribution $\mathcal N(0,τ^2)$. Then do the following $n$ times: - Randomly select a $\mu$ from the $\mu$s picked before, and randomly select an $x$ from the distribution $\mathcal N(u,σ^2)$.

By the way, this is from a tutorial on variational inference. See bottom of page 1.



Think about it like the following:

$z_i \sim Mult(\pi)$ where $\pi$ is a $\textit{vector}$ of probabilities of size $K$. The multinomial distribution is a generalization of the categorical distribution, which the poster below correctly mentions. The categorical distribution is equivalent to considering the multinomial when we have one trial and more than 2 categories for consideration. In other words, the author of the document could have replace the multinomial specification with the categorical and would have been more easily interpreted.

As a very concrete example, let's consider the case where you have just completed step (1) above and have drawn all the means, $\mu_k$ for $k = 1, \ldots, K$. Now imagine we have a $K$-sided die that you can roll, where each side has probability $\pi_k$ of being chosen for $k = 1, \ldots, K$. (Normally each side of a dice roll is equally likely but here they may not be equal). These $\pi_k$ can be collected into a vector $\pi = (\pi_{1}, \ldots, \pi_K)$, which is what we put into the "Mult" specification in part (a) above. Then, the "action":

$$ z _ { i } \sim \operatorname { Mult } ( \pi ) $$

is the same as rolling the $K$-sided die and setting $z_i$ equal to the result of of the dice roll. Here $z_i$ can take values in the range of $\{1, \ldots, K\}$. Then, $z_i$ will serve as the index of which of the means $\mu_k$ you had generated earlier will be used to now serve as the mean of the sampling distribution of $x_i$. Now repeat this process for each $i \in \{1, \ldots, n\}$.

Finally, let's consider a real-life example. Suppose $K=3$ and $n=2$, then we will first in step (1) draw out three means $\mu_1, \mu_2, \mu_3$. Then, we have a 3-sided die for which we may roll a $1$ with probability $\pi_1$, a $2$ with probability $\pi_2$, and a $3$ with probability $\pi_3$. Then, $\pi = (\pi_1, \pi_2, \pi_3)$. Then we want to do parts (a) and (b) twice since we set $n=2$.

Roll the 3-sided die and record what we got. Let's say for the $n=1$ case we rolled a $2$. Then, we set $z_1 = 2$. For the $n=2$ case let's say we rolled a $3$. Then we set $z_2 = 3$. Now, we have:

Take the $z_1$ and $z_2$ values to serve as the index of which of the three means we want to use for sampling $x_1$ and $x_2$. This entails sampling:

$$ x _ { 1 } \sim \mathcal { N } \left( \mu _ { 2 } , \sigma ^ { 2 } \right) $$


$$ x _ { 2 } \sim \mathcal { N } \left( \mu _ { 3 } , \sigma ^ { 2 } \right) $$

and plugging in the generated $\mu_2$ and $\mu_3$ from part (1) above, since $\mu_{z_1} = \mu_{2}$ and $\mu_{z_2} = \mu_{3}$.

  • 1
    $\begingroup$ There was a bug with stackexchange, I've reposted my answer above. $\endgroup$ – user321627 Feb 8 at 8:25
  • $\begingroup$ very clear. thanks! $\endgroup$ – AlphaBeta Feb 8 at 17:14
  • $\begingroup$ You're welcome, if you feel this has helped please upvote or select as the main answer! $\endgroup$ – user321627 Feb 9 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.