The information inequality is a theorem that shows that the Kullback-Leibler divergence between two probability distributions is always non negative. This can be proved easily using the Jensen's inequality with the $-\log$ function, but the proofs I have read always have to distinguish if the probability distributions are defined by continuous or discrete random variables.

I was trying to see if it is possible to not distinguish cases, and I thinked about using measure theory, with the Jensen's inequality provided in Rudin's book:

Let $\mu$ be a probability measure on a $\sigma$-algebra $\mathcal{M}$ in a set $\Omega$. If $f$ is a real integrable function with $a < f(x) < b$ for all $x \in \Omega$, and if $\varphi$ is convex on $]a,b[$, then $$\varphi\left( \int_{\Omega}f d\mu\right) \le \int_{\Omega}(\varphi \circ f) d\mu. $$

So using this I have $$KL(p\|q) = \int\log\frac{p(x)}{q(x)}dp = \int - \log \frac{q(x)}{p(x)} dp \ge - \log \int\frac{q(x)}{p(x)} dp, $$ but now the only way I find to continue is distinguishing if the distributions are discrete or continuous. I haven't studied measure theory and I only have some basic notions, so I'm not sure how to continue. Also, if my notations are wrong, please tell me. Any help will be appreciated.

  • 1
    $\begingroup$ Your derivation immediately generalized to the case where $q$ is absolutely continuous w.r.t. $p$, meaning that $dq(x)=f(x)dp(x)$ for some measurable function $f\geq0$. $\endgroup$ – Sangchul Lee Aug 17 '18 at 9:56

A rigorous definition of information theory quantities can be found in M.S. Pinsker's book "Information and information stability of random variables and processes".

Consider two probability measures $\mu$ and $\nu$ defined on the measurable space $(\Omega,\mathcal M)$. Let $\{E_i\}$ be a partition of $\Omega$. Then KL-divergence can be defined as $$ D(\mu\|\nu)=\sup\sum_i \mu(E_i)\log\frac{\mu(E_i)}{\nu(E_i)} $$ where the supremum is taken over all partitions of $\Omega$. Since this definition utilizes a sum like discrete KL-divergence, the non-negative property follows accordingly: $$ D(\mu\|\nu)\geq 0. $$

There is a theorem by Gelfand-Yaglom-Perez stating that if $D(\mu\|\nu)$ is finite, then $\mu$ is absolutely continuous with respect to $\nu$ and $$ D(\mu\|\nu)=\int_\Omega \log\frac{d\mu}{d\nu}\mathrm{d}\mu, $$ where $\frac{d\mu}{d\nu}$ is the Radon-Nikodym derivation. If you want, you can define the KL-divergence directly using the above equation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.