Show that relative entropy in terms of integrals is always non-negative

Question

I'm looking to prove that relative entropy (or Kullback–Leibler divergence) is always non-negative (i.e. Gibbs' inequality) in terms of integrals, that is show that for any two probability PDF's $P$ and $Q$ in $\Omega$

$$ \int_\Omega P(x) \log \left(\frac{P(x)}{Q(x)}\right) \, \mathrm{d}x \geq 0. $$

I know how to show this with Jensen's inequality, but I'm having trouble showing that if the integral in question diverges. As an example, setting $P(x)=\frac{1}{\pi \left(x^2+1\right)}$, $Q(x)=\frac{e^{-\frac{x^2}{2}}}{\sqrt{2 \pi }}$ and $\Omega=\mathbb{R}$, the integral diverges to $\infty$. In this case, as I'm only trying to show that the integral can never be negative, it's enough to show the integral can only diverge to positive infinity. While this seems heuristically correct, I don't know how to go on proving that.

My first approach was to look at the integrand itself, but it can be negative at times. Also, I can't just look at some cutoff value from which on all the values would be positive, as I'm only guaranteed that

$$\lim_{x\to \infty} \frac{P(x)}{Q(x)}>0,$$

which still allows for negative values inside the logarithm.

Any ideas or references to some literature? All the proofs I've found seem to ignore the possibility of divergence, maybe there's a reason I could do that too?

Answer 1

Take $x$ such that $P(x)$ is non-zero. Observe that if $Q(x)$ is also non-zero, then \begin{align} \log\frac{P(x)}{Q(x)}=-\log\frac{Q(x)}{P(x)}\geq 1-\frac{Q(x)}{P(x)}\;, \end{align} and thus $P(x)\log\frac{P(x)}{Q(x)}\geq P(x)-Q(x)$. If on the other hand $Q(x)=0$, then $P(x)\log\frac{P(x)}{Q(x)}=+\infty$ (by convention, justified by continuity), which is again $\geq P(x)-Q(x)$.

Put together, it follows that \begin{align} \int P(x)\log\frac{P(x)}{Q(x)}\mathrm{d}x \end{align} exists and is larger than or equal to \begin{align} \int \big[P(x)-Q(x)\big]\mathrm{d}x &= 0 \;. \qquad\square \end{align}