# How does generative model work? Especially in the Natural Language Processing?

AFAIK, the generative language model consists of a probability distribution for some vocabulary. I am wondering how to use this probability distribution to generate a stream of words, i.e. language?

If I always pick the word with biggest probability, it will always be the same word because the distribution is fixed.

I am not sure if I understand it correctly. Could anyone provide a concrete operational example? Maybe with a toy distribution.

Or maybe I can put the question this way:

How to generate a sequence of outcomes given the distribution of a discrete random variable?

-

I'd sooner post this as a comment — this is a question I've long meant to get better acquainted with but simply haven't had the time — but it won't fit there.

First, if you (or, more likely, a computer) always pick the word with the greatest probability, then it's not a random process. But even if the process is going about in some random fashion, it won't always pick the most likely outcome: A biased coin still lands on the less-likely face from time to time.

If you had enough data to help you describe a multinomial distribution for the words in, say, English, that would enable you to output text with words in (more or less) the right proportion — the accounting for 7 percent of the output, of 3 percent, and so on. However, this approach pays no attention to the order of the words. (The most likely two-word phrase would be the the.)

Instead, a Markov model can be used to assign probabilities to bits of text based on what bits of text immediately precede them. With such an approach, you look at $n$ characters or words and determine the probability of the next one; assembling the distribution for each $n$-gram can be a bit arduous, though, if you're not using a computer. But once that information has been encoded, you can randomly select the first word; say it comes up the. Then, based on that first word, the process will randomly select the second according to the distribution of words that follow the; we'd expect here that the distribution would be more likely to pick a noun than, say, a verb.

In the above example, $n = 1$; that is, your $n$-gram is of length 1. But you could start with two words (or however many you want); the process would then find the next two-word phrase that starts with the second word in your initial phrase. (To use the terminology, a Markov model assigns a probability to the movement from one state to another; for example, given the phrase the baddest, we might sooner expect to move to baddest dude than to baddest sculpture.)

And it continues from there. The problem is that, depending on how you choose your unit, you might need to deal with a lot of information. So, do you choose to look at strings of $n$ characters or strings of $n$ words? We could probably get by with, say, 30 characters (if we ignore numbers and include, say, a space, a period, etc.), but how many words suffice? Apparently, 135 words can get you half the Brown Corpus, but it's really your source text (the text from which you determine the distributions) that will determine which words are used in your model. Let's say your source text includes 200 distinct words. Looking at some example output, you might decide that you need strings of four words to make something looks sufficiently English; using characters as your unit, you find that strings of length seven are necessary. These numbers create upper bounds for the number of states you'd have to account for: In this case, $200^4 = 1.6 \times 10^9 < 30^7 = 2.187 \times 10^{10}$. Obviously, most of these combinations will not be used; the Brown Corpus contains just $10^6$ words, and your source text probably doesn't contain strings like iudddzx. But however many states you come up with — say it's $k$ — you'll need a big ($k\times k$) matrix to compute the probabilities of moving between them.

I found this page a while back and thought it a pretty good intro. It includes, I guess, what you might call toy distributions: There are some bits of text that are used as input for determining how to construct language, along with the output that each created. The results are, perhaps, more humorous than you might find practical, but... Maybe you'd just need a larger $n$?

-