# Derivation 9.97 in Jaynes' Probability Theory

In page 298 of Jaynes' Probability Theory: the Logic of Science, equation (9.97), Jaynes says:

We expect that, if hypothesis $H$ is true, then $n_k$ will be close to $np_k$, in the sense that the difference $|n_k-np_k|$ will grow with $n$ only as $\sqrt n$. Call this 'condition A'. Then using the expansion $\log(x) = (x-1)-(x-1)^2/2+...$, we find that

$$\sum_{k=1}^mn_k\log\left[\frac{n_k}{np_k}\right] = \frac 1 2\sum_k\frac{(n_k-np_k)^2}{np_k} + O\left(\frac 1 {\sqrt n}\right)$$

In the above, $\sum_kp_k = 1$ and $\sum_kn_k=n$, where $n$ is the total number of trials in a series of Bernoulli trials and $n_k$ is the number of trials that had outcome $k$.

I'd like to know how he got from that expansion to the quoted equation, especially given that the squared term in the expansion has negative sign.

Jaynes' nomenclature confused me a bit. He calls hypotheses of the type "there are $m$ possible outcomes of an experiment, each being observed with probability $p_k$ independent of previous or future repetitions of that experiment" the "Bernoulli class." $n_k$ is the number of performed experiments that had outcome $k$, $n$ is the total number of experiments performed. But that is all that's defined by Jaynes, and all that's known about the hypotheses.

(A thought occurs: what if Jaynes meant that condition A is that $|n_k -np_k| \approx O(1/\sqrt n)$? I haven't explored this possibility to know whether it makes sense.)

-
I don't understand "$n_k$ is the number of Bernoulli trials with outcome $k$". A Bernoulli trial has outcomes 0 and 1. Do you mean $n_k$ is a binomial distribution with parameters $n$ and $p_k$? But that would contradict $\sum_k n_k=n$. –  Teepeemm May 7 at 17:19

$\sum_k (n_k-np_k)^2/np_k$ is Pearson's chi-squared statistic. The left side of the equation is the log likelihood ratio of the mle for the multinomial distribution vs the probabilities under the null hypothesis, the $p_k$'s. A common result is that 2*log likelihood ratio (likelihood alternative/likelihood null) has a chi-squared distribution under the null hypothesis with df = k-1. It is also well known that the Pearson Chi-squared statistic for the goodness of fit test also has k-1 df. The left and right sides of the equation have the same asymptotic distribution under the null hypothesis. The proof of either of these results should help.

As a side note, the meaning of $n_k-np_k$ growing with rate $\sqrt{n}$ could mean that since $n_k \sim binomial(n, p_k)$,

$\frac{1}{\sqrt{n}}(n_k-np_k) \overset{d}{\to} N(0, p_k(1-p_k)) \text{ as } n\to \infty$, the standard normal approximation to the binomial.

Or it could mean that $\vert n_k-np_k \vert = O_p(\sqrt{n}) \text{ or } O(\sqrt{n})$

-
could you expand a bit? –  nbubis May 9 at 7:22
@nbubis I could indeed, but probably not until tomorrow. Which part do you want expanded? –  jsk May 9 at 7:36
Indeed that's Pearson's chi-squared statistic, and the point of this passage of the book is exactly to "prove" that the log-likelihood ratio (which Jaynes calls psi) is approximately proportional to the chi-squared under certain circumstances, and that's exactly the proof that I don't understand x) And if you could replicate and/or point to the proof that they have the same asymptotic distribution, that'd help, though then I'd be wondering how the hell Jaynes came up with that log series expansion thing. –  Pedro Carvalho May 9 at 11:44
@PedroCarvalho check out the proof here... unc.edu/courses/2006spring/ecol/145/001/docs/lectures/… –  jsk May 9 at 19:21
That's great, thanks! –  Pedro Carvalho May 10 at 2:37

The best I can get is \begin{aligned} \sum_{k=1}^m n_k\log\left[\frac{n_k}{n p_k}\right] & = \sum_{k=1}^m n_k\left[\left(\frac{n_k}{n p_k}-1\right)-\frac12\left(\frac{n_k}{n p_k}-1\right)^2+O\left(\left(\frac{n_k}{n p_k}-1\right)^3\right)\right] \\ & = \sum_{k=1}^m n_k\left[-\frac12\left(\frac{n_k}{n p_k}\right)^2+\frac{2n_k}{n p_k}-\frac32+O\left(\left(\frac{n_k-n p_k}{n p_k}\right)^3\right)\right] \\ & = \sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}-\frac{3n_k}2+n_k O\left(\left(\frac{\sqrt n}{n p_k}\right)^3\right) \qquad\text{condition A} \\ & = \sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}-\frac{3n_k}2+n_k O\left(n^{-3/2}\right) \\ & = \left[\sum_{k=1}^m -\frac{n_k^3}{2n^2 p_k^2}+\frac{2n_k^2}{n p_k}\right]-\frac{3n}2+O\left(n^{-1/2}\right) \end{aligned}

But something isn't right. Your desired expression is $O(1)$, while the expansion appears to be $O(\sqrt n)$: condition A says $n_k=np_k+O(\sqrt n)$, so that the left hand side is $$\sum_k(np_k+O(\sqrt n))\log(1+O(n^{-1/2})=\sum_k(np_k+O(\sqrt n))O(n^{-1/2}).$$ I think we need to use something about the definition of $p_k$, $n_k$, and $n$.

-
I have edited the original post to add more information. –  Pedro Carvalho May 9 at 3:11