I'm reading the Latent Dirichlet Allocation paper trying to understand it. However I got stuck at the very first part! When they sampled from a multinomial distribution and considered the result to be a single number! My understanding if the multinomial distribution is to return a vector where the number of its elements are the same as the number of probability vector parameter plugged into the multinomial.

I'm not sure if I misunderstood the notation.

Here is the link to the paper $\text{(page $4$)}$: http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf

$\mathbf 3$. Latent Dirichlet allocation

Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words.$^1$
$\quad$LDA assume the following generative process for each document $\mathbf w$ in a corpus $D$:

  1. Choose $N\sim\operatorname{Poisson}(\xi)$.

  2. Choose $\theta\sim\operatorname{Dir}(\alpha).$

  3. For each of the $N$ words $w_n$:

    • $\text a.$ Choose a topic $z_n\sim\operatorname{Multinomial}(\theta)$.
    • $\text b.$ Choose a word $w_n$ from $p(w_n|z_n,\beta)$, a multinomial probability conditioned on the topic $z_n$.

A multinomial distribution is a probability distribution on a vector-valued random variable.

That said, from what I can tell from the paper, "words" and "topics" are vectors, not scalars. A "word" consists of a $(0,1)$-valued vector. The length of the vector is the size of the set of all words. While it is not explicitly described in the paper, a "topic" is implied to be a vector, whose length is the same as that for words. I think a topic can be regarded as a collection of words; i.e., it is a vector for which more than one position can be equal to $1$ (unlike a word, for which exactly one position is $1$ and the others are all $0$).

  • $\begingroup$ I don't really understand it. A multinomial distribution accepts a vector of scalars (let's say [3, 1, 5], where each scalar is the number of successes of a category) and returns the probability that this combination of successes happen. So if I sample from such a distribution I should get a vector [k1,k2,k3]. I am true? $\endgroup$ – Jack Twain Jan 4 '14 at 14:46
  • $\begingroup$ What I am sure about is that Z is a single scalar and theta is a real-valued vector. $\endgroup$ – Jack Twain Jan 4 '14 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.