Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In page 298 of Jaynes' Probability Theory: the Logic of Science, equation (9.97), Jaynes says:

We expect that, if hypothesis $H$ is true, then $n_k$ will be close to $np_k$, in the sense that the difference $|n_k-np_k|$ will grow with $n$ only as $\sqrt n$. Call this 'condition A'. Then using the expansion $\log(x) = (x-1)-(x-1)^2/2+...$, we find that

$$\sum_{k=1}^mn_k\log\left[\frac{n_k}{np_k}\right] = \frac 1 2\sum_k\frac{(n_k-np_k)^2}{np_k} + O\left(\frac 1 {\sqrt n}\right)$$

In the above, $\sum_kp_k = 1$ and $\sum_kn_k=n$, where $n$ is the total number of trials in a series of Bernoulli trials and $n_k$ is the number of trials that had outcome $k$.

I'd like to know how he got from that expansion to the quoted equation, especially given that the squared term in the expansion has negative sign.

--EDIT to add info--

Jaynes' nomenclature confused me a bit. He calls hypotheses of the type "there are $m$ possible outcomes of an experiment, each being observed with probability $p_k$ independent of previous or future repetitions of that experiment" the "Bernoulli class." $n_k$ is the number of performed experiments that had outcome $k$, $n$ is the total number of experiments performed. But that is all that's defined by Jaynes, and all that's known about the hypotheses.

(A thought occurs: what if Jaynes meant that condition A is that $|n_k -np_k| \approx O(1/\sqrt n)$? I haven't explored this possibility to know whether it makes sense.)

share|cite|improve this question
I don't understand "$n_k$ is the number of Bernoulli trials with outcome $k$". A Bernoulli trial has outcomes 0 and 1. Do you mean $n_k$ is a binomial distribution with parameters $n$ and $p_k$? But that would contradict $\sum_k n_k=n$. – Teepeemm May 7 '14 at 17:19
up vote 1 down vote accepted

$\sum_k (n_k-np_k)^2/np_k$ is Pearson's chi-squared statistic. The left side of the equation is the log likelihood ratio of the mle for the multinomial distribution vs the probabilities under the null hypothesis, the $p_k$'s. A common result is that 2*log likelihood ratio (likelihood alternative/likelihood null) has a chi-squared distribution under the null hypothesis with df = k-1. It is also well known that the Pearson Chi-squared statistic for the goodness of fit test also has k-1 df. The left and right sides of the equation have the same asymptotic distribution under the null hypothesis. The proof of either of these results should help.

As a side note, the meaning of $n_k-np_k$ growing with rate $\sqrt{n}$ could mean that since $n_k \sim binomial(n, p_k)$,

$\frac{1}{\sqrt{n}}(n_k-np_k) \overset{d}{\to} N(0, p_k(1-p_k)) \text{ as } n\to \infty $, the standard normal approximation to the binomial.

Or it could mean that $\vert n_k-np_k \vert = O_p(\sqrt{n}) \text{ or } O(\sqrt{n})$

share|cite|improve this answer
could you expand a bit? – nbubis May 9 '14 at 7:22
@nbubis I could indeed, but probably not until tomorrow. Which part do you want expanded? – jsk May 9 '14 at 7:36
Indeed that's Pearson's chi-squared statistic, and the point of this passage of the book is exactly to "prove" that the log-likelihood ratio (which Jaynes calls psi) is approximately proportional to the chi-squared under certain circumstances, and that's exactly the proof that I don't understand x) And if you could replicate and/or point to the proof that they have the same asymptotic distribution, that'd help, though then I'd be wondering how the hell Jaynes came up with that log series expansion thing. – Pedro Carvalho May 9 '14 at 11:44
@PedroCarvalho check out the proof here...… – jsk May 9 '14 at 19:21
That's great, thanks! – Pedro Carvalho May 10 '14 at 2:37

The best I can get is $$ \begin{aligned} \sum_{k=1}^m n_k\log\left[\frac{n_k}{n p_k}\right] & = \sum_{k=1}^m n_k\left[\left(\frac{n_k}{n p_k}-1\right)-\frac12\left(\frac{n_k}{n p_k}-1\right)^2+O\left(\left(\frac{n_k}{n p_k}-1\right)^3\right)\right] \\ & = \sum_{k=1}^m n_k\left[-\frac12\left(\frac{n_k}{n p_k}\right)^2+\frac{2n_k}{n p_k}-\frac32+O\left(\left(\frac{n_k-n p_k}{n p_k}\right)^3\right)\right] \\ & = \sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}-\frac{3n_k}2+n_k O\left(\left(\frac{\sqrt n}{n p_k}\right)^3\right) \qquad\text{condition A} \\ & = \sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}-\frac{3n_k}2+n_k O\left(n^{-3/2}\right) \\ & = \left[\sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}\right]-\frac{3n}2+O\left(n^{-1/2}\right) \end{aligned} $$

But something isn't right. Your desired expression is $O(1)$, while the expansion appears to be $O(\sqrt n)$: condition A says $n_k=np_k+O(\sqrt n)$, so that the left hand side is $$\sum_k(np_k+O(\sqrt n))\log(1+O(n^{-1/2})=\sum_k(np_k+O(\sqrt n))O(n^{-1/2}).$$ I think we need to use something about the definition of $p_k$, $n_k$, and $n$.

share|cite|improve this answer
I have edited the original post to add more information. – Pedro Carvalho May 9 '14 at 3:11

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.