Let's say I'm defining a LDA as the following:

  • For each doc $m$:
    • Sample topic probabilities $\theta_m \sim Dirichlet(\alpha)$
    • For each word $n$:
      • Sample a topic $z_{mn} \sim Multinomial(\theta_m)$
      • Sample a word $w_{mn} \sim Multinomial(\beta)$

where $\alpha, \beta$ are fixed hyperparameters.

Now, to do this, I'm going to use an EM algorithm. For the 'E' step, given $\alpha$ and $\beta$, I infer $z_{mn}$ for all $n$ and $\theta_m$ for all $m$ given $w, \alpha, \beta$ using Gibbs sampling. Equivalently, I'm trying to find $p(z, \theta | w, \alpha, \beta) = \frac{p(z, \theta, w | \alpha, \beta)}{p(w|\alpha, \beta)}$ using Gibbs sampling.

At a high level, I understand what Gibbs Sampling is and what this LDA model does. We assigns topics to each word and document (assuming all others are correct). We do repeat this process in a chain until we maximize the probability of each $m$ and $n$ belonging to a particular category.

However, I'm having trouble representing the conditional probabilities of the Gibbs Sampler for this model. Where do I even start and what am I looking for when asked to find the conditional probabilities of this sampler?


1 Answer 1


If you are finding it tricky getting started, I suggest you have a look at this paper by Griffiths and Steyvers here. It was well-known in the ML community for using Gibbs sampling as an alternative to variational inference for topic models. If you are looking for verbal exposition, have a look at the slides and video at around the 40:00 min mark of Lecture 14 on MCMC of this series here. The instructor briefly walks through Gibbs sampling algorithmically and in context of the paper above, and also briefly provides intuition on one of the equations there.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .