Suppose we have a multinomial distribution with $N$ possible outcomes, with probabilities $p_1,\ldots,p_N$. We sample this $n$ times, and denote the observed frequency of the $i$th outcome as $\xi_i$. In [1] the author claims that the distribution of the $\xi_i$ in the limit of large $n$ is:


We can see immediately that this must be an approximation, as it assigns nonzero probabilities for $\xi_1+\cdots+\xi_N>1$. However we can see that these have vanishing probability in the limit $n\rightarrow\infty$. My question is how do we derive (1) from the multinomial distribution, and show that they match in the $n\rightarrow\infty$ limit?

My thoughts

My first thought would be to appeal to the central limit theorem. The multinomial distribution has mean $\mu_i=p_i$ and covariance matrix $\Sigma_{ij}=\delta_{ij}p_i-p_ip_j$, so we would expect this in the large $n$ limit to be described by a multivariate Gaussian with mean $\mu$ and covariance $\frac{1}{n}\Sigma$. However, things are complicated by the fact that the multinomial covariance is singular (since $\xi_N$ is determined by the other $\xi_i$s), and so the multivariate Gaussian is not defined.

To address this, we may try and consider only the first $\xi_1,\ldots,\xi_{N-1}$, which have a non-singular covariance matrix and hence well-defined multivariate Gaussian distribution. Let's take the Binomial distribution $N=2$. The frequency $\xi_1$, this has mean $p_1$ and variance $p_1(1-p_1)$, so this would be described the the Gaussian: $$\propto\exp\left(-\frac{n}{2}\frac{(\xi_1-p_1)^2}{p_1(1-p_1)}\right).\;\;\;\;\;(2)$$ The expression (1) gives: $$\propto\exp\left(-\frac{n}{2}\left(\frac{(\xi_1-p_1)^2}{p_1}+\frac{(\xi_2-p_2)^2}{p_2}\right)\right).\;\;\;\;\;(3)$$ If we substitute $\xi_2\rightarrow 1-\xi_1$, $p_2\rightarrow 1-p_1$ into (3), we can verify that this gives the same answer as (2). I have verified that this also works for $N=4$.

I'm sure that if I just bashed out the algebra for general $N$ we would get agreement between the central limit theorem and (1) when we restrict the latter to $\xi_1+\cdots+\xi_N=1,p_1+\cdots+p_N=1$. However, how can we start with the multinomial distribution and derive (1) as a limit which is valid everywhere? One idea would be to say that (1) goes to zero as $n\rightarrow\infty$ when you are not on that plane, however I am a bit uncomfortable with this as it goes to zero everywhere except the mean as $n\rightarrow\infty$, so I don't know if that argument is good enough.

[1] Wootters, William K. "Statistical distance and Hilbert space." Physical Review D 23.2 (1981): 357.


I ended up solving this by generalising the proof in Gnedenko - The Theory of Probability of the DeMoivre-Laplace theorem in Section II, which is basically just this result in the binomial case. For future reference in case anyone stumbles across this question the proof is below, apologies for not LaTeXing this.

Since the probability of any given $m_i$ ends up in the tail and goes to zero as $n\rightarrow\infty$, the strategy is to instead change variables to $x_i$, the number of standard deviations from the mean, which we would intuitively expect to be Gaussian distributed. We hold $x_i$ constant, and then let $n\rightarrow\infty$.The $x_i$ are assumed uniformly bounded above and below, however this isn't a problem as we have convergence for any arbitrary bound.

Note that $q_i$ is defined as $1-p_i$.

enter image description here enter image description here enter image description here enter image description here

  • $\begingroup$ What is $q_i$? Is it $q_i=1-p_i$? $\endgroup$ – Carlos Pinzón Mar 9 at 18:24
  • $\begingroup$ Yes, that's correct. Thanks for pointing that out, I will amend the answer. $\endgroup$ – Ruvi Lecamwasam Mar 10 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.