1
$\begingroup$

I am getting a weird result here:

Let $p_1 = q_1 + \Delta$ and Let $p_2 = q_2 - \Delta$

I use the expansion $\log(1-x) = -x -x^2/2 -x^3/3 - ...$ in the third step. This expansion is valid for $x<1$ so there should be no problem with that.

\begin{align*} D(\vec{p}||\vec{q}) &= p_1\log{\frac{p_1}{q_1}} + p_2\log{\frac{p_2}{q_2}}\\ &= p_1\log{\frac{q_1 + \Delta}{q_1}} + p_2\log{\frac{q_2 - \Delta}{q_2}}\\ &= p_1(\frac{\Delta}{q_1} - \frac{\Delta^2}{q_1^2} + \frac{\Delta^3}{q_1^3} - ...) + p_2(-\frac{\Delta}{q_2} - \frac{\Delta^2}{q_2^2} - \frac{\Delta^3}{q_2^3} - ...)\\ &= (q_1 + \Delta)(\frac{\Delta}{q_1} - \frac{\Delta^2}{q_1^2} + \frac{\Delta^3}{q_1^3} - ...) + (q_2 - \Delta)(-\frac{\Delta}{q_2} - \frac{\Delta^2}{q_2^2} - \frac{\Delta^3}{q_2^3} - ...)\\ &=0 \end{align*}

The last equality follows from comparing terms of powers of $\Delta$ (am I not allowed to do that?).

$\endgroup$
0
$\begingroup$

There are at least two errors: $$p_1\log{\frac{q_1 + \Delta}{q_1}} = p_1\left(\frac{\Delta}{q_1} - \frac{\Delta^2}{\color{red}{2}q_1^2} - \frac{\Delta^3}{\color{red}{3}q_1^3} - \cdots\right)$$ Similar for the second term.

And even if the low order terms cancel, the result is not zero. Using Maple I get (if I did not make mistakes) $$D(\vec{p}||\vec{q}) = \frac{1}{2}\left(\frac{1}{q_1}+\frac{1}{q_2}\right) \Delta^2 + O(\Delta^3)$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Oh I am so sorry. Stupid mistake. $\endgroup$ – Abhijeet Melkani Jul 4 '18 at 10:31
1
$\begingroup$

As you wrote, $$\log(1-x)\sim -x-\frac12x^2-\frac13x^3-\dots\ne-x-x^2-x^3-\dots$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am so sorry. Stupid mistake. $\endgroup$ – Abhijeet Melkani Jul 4 '18 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.