Probability a given probability ditribution was used Consider a set of $k$ different discrete probability distributions on a finite set. One of these distributions is chosen uniformly at random and a lot of elements are sampled from this distribution. I am given these samples and the $k$ different distributions. How can I compute or estimate (preferably quickly) which of these distributions is the most likely to have been chosen?  This is a real world problem.
 A: Let's consider two distributions over the set $\{H, T\}$ (for "heads" and "tails"):


*

*distribution 1: $H$ with probability 1/2, $T$ with probability 1/2.

*distribution 2: $H$ with probability 2/3, $T$ with probability 1/3.


You have 100 observations; they are 60 Hs and 40 Ts.  Which is more likely: distribution 1 or distribution 2?  For concreteness, we'll say that you got the 60 Hs first, followed by the 40 Ts; the arithmetic works out the same for any order.
Then you want to compute a conditional probability $P(H^{60} T^{40} | D_1) = (1/2)^{100} \approx 7.88 \times 10^{-31}$ - that is, if you're working from distribution 1, the probability of getting 60 heads followed by 40 tails is $(1/2)^{100}$. (This probability is the same no matter which 100 coin flip outcomes you have.)
Similarly, $P(H^{60} T^{40} | D_2) = (2/3)^{60} (1/3)^{40} \approx 2.23 \times 10^{-30}$.  Since this probability is larger, 60 heads and 40 tails is evidence in favor of selecting from disribution 2.
If you're worried about the computations underflowing, you can take logs of the probabilities, so here you'd ger
$$\log P(H^{60} T^{40} | D_1) = 100 \log (1/2) \approx -69.31 $$
and
$$\log P(H^{60} T^{40} | D_2) = 60 \log (2/3) + 40 \log (1/3) \approx -68.27$$
and the second log-probability is larger.
Since you're assuming that $D_1$ and $D_2$ are equally likely before anything is observed, you actually have a prior and can compute by Bayes' theorem
$$ P(D_1 | H^{60} T^{40}) = {P(H^{60} T^{40} \hbox{ and } D_1) \over P(H^{60} T^{40})} = {P(H^{60} T^{40} | D_1) P(D_1) \over P(H^{60} T^{40} | D_1) P(D_1) + P(H^{60} T^{40} | D_2) P(D_2)} $$
and putting in the numerical values from above and $P(D_1) = P(D_2) = 1/2$ gives you
$$ {(1/2)^{100} \times (1/2) \over (1/2)^{100} \times (1/2) + (2/3)^{60} (1/3)^{40} \times (1/2)} \approx 0.2607$$
This will generalize to a larger discrete set, more distributions, etc.
