The information inequality is a theorem that shows that the Kullback-Leibler divergence between two probability distributions is always non negative. This can be proved easily using the Jensen's inequality with the $-\log$ function, but the proofs I have read always have to distinguish if the probability distributions are defined by continuous or discrete random variables.

I was trying to see if it is possible to not distinguish cases, and I thinked about using measure theory, with the Jensen's inequality provided in Rudin's book:

Let $\mu$ be a probability measure on a $\sigma$-algebra $\mathcal{M}$ in a set $\Omega$. If $f$ is a real integrable function with $a < f(x) < b$ for all $x \in \Omega$, and if $\varphi$ is convex on $]a,b[$, then $$\varphi\left( \int_{\Omega}f d\mu\right) \le \int_{\Omega}(\varphi \circ f) d\mu. $$

So using this I have $$KL(p\|q) = \int\log\frac{p(x)}{q(x)}dp = \int - \log \frac{q(x)}{p(x)} dp \ge - \log \int\frac{q(x)}{p(x)} dp, $$ but now the only way I find to continue is distinguishing if the distributions are discrete or continuous. I haven't studied measure theory and I only have some basic notions, so I'm not sure how to continue. Also, if my notations are wrong, please tell me. Any help will be appreciated.

  • 1
    $\begingroup$ Your derivation immediately generalized to the case where $q$ is absolutely continuous w.r.t. $p$, meaning that $dq(x)=f(x)dp(x)$ for some measurable function $f\geq0$. $\endgroup$ Aug 17, 2018 at 9:56

1 Answer 1


A rigorous definition of information theory quantities can be found in M.S. Pinsker's book "Information and information stability of random variables and processes".

Consider two probability measures $\mu$ and $\nu$ defined on the measurable space $(\Omega,\mathcal M)$. Let $\{E_i\}$ be a partition of $\Omega$. Then KL-divergence can be defined as $$ D(\mu\|\nu)=\sup\sum_i \mu(E_i)\log\frac{\mu(E_i)}{\nu(E_i)} $$ where the supremum is taken over all partitions of $\Omega$. Since this definition utilizes a sum like discrete KL-divergence, the non-negative property follows accordingly: $$ D(\mu\|\nu)\geq 0. $$

There is a theorem by Gelfand-Yaglom-Perez stating that if $D(\mu\|\nu)$ is finite, then $\mu$ is absolutely continuous with respect to $\nu$ and $$ D(\mu\|\nu)=\int_\Omega \log\frac{d\mu}{d\nu}\mathrm{d}\mu, $$ where $\frac{d\mu}{d\nu}$ is the Radon-Nikodym derivation. If you want, you can define the KL-divergence directly using the above equation.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .