Let $P_n (x,y)$ be a sequence of (cumulative) probability distributions defined on $\mathcal{X}\times \mathcal{Y}$ (of arbitrary cardinality), that weakly converges to $P(x,y)$:

$$ P_n (x,y) \Rightarrow P(x,y) $$

i.e., $P_n (x,y)$ converges pointwise to $P(x,y)$, except on the points of discontinuity of $P(x,y)$.

Let $I_n$ be the mutual information induced by $P_n (x,y)$, defined by

$$ I_n = \int \log\frac{dP_n(x,y)}{d\left( P_n(x) \times P_n(y)\right)} dP_n $$

where $\frac{dP_n(x,y)}{d\left( P_n(x) \times P_n(y)\right)}$ is the Radon-Nikodym derivative of the joint distribution with respect to the product of the marginals.

Similarly, define

$$ I = \int \log\frac{dP(x,y)}{d\left( P(x) \times P(y)\right)} dP $$

Is it in general true that $I_n \to I$ as $n \to \infty$? If not, what are the necessary and/or sufficient conditions under which this holds true?

I feel like measure theoretic convergence theorems (bounded convergence, monotone convergence, dominated convergence) might be useful in approaching this problem, but I could not find a way of directly applying them.

  • $\begingroup$ So the marginals $P_n(x), P_n(y)$ also vary with $n$? $\endgroup$ – A Blumenthal Oct 2 '14 at 16:24
  • $\begingroup$ Yes, $P_n(x),P_n(y)$ are the marginals corresponding to the joint distribution $P_n(x,y)$, which can vary arbitrarily, with the condition that they weakly converge to $P(x),P(y)$, respectively. $\endgroup$ – karakusc Oct 2 '14 at 19:38
  • $\begingroup$ I really doubt this is true, I'll try and come up with a counterexample later. $${}$$ Mutual information is a KL divergence, which is fundamentally a lot more sensitive than things that a weak topology can describe. KL divergence cares a lot about small-probability events in $\hat{P}$, and penalizes you heavily when these events turn out to be slightly higher probability under $P$. By the way it is defined, weak convergence does not make this strong of statements. Everyone wants to do what you are trying, but I suspect in reality it will take some pretty heavy assumptions on $P$. $\endgroup$ – Christian Chapman May 13 '16 at 6:36

This does not necessarily hold. As an example, let $Z$ be a random variable, with $H(Z) > 0$. Consider the sequence $(X_n,Y_n) = \left(\frac{1}{n}Z,\frac{1}{n}Z\right)$. Then, $(X_n,Y_n) \xrightarrow[\text{}]{\text{d}} (0,0)$ as $n\to\infty$, while

$$ I(X_n;Y_n)=I(Z;Z) = H(Z) > 0 = I(0;0). $$


In general, mutual information is weak lower-semicontinuous, i.e. if $(X_n,Y_n)$ converges weakly (in distribution) to $(X,Y)$, then $$ I(X;Y) \leq \liminf_{n\to\infty}I(X_n;Y_n). $$


To prove lower semicontinuity of the divergence, we will do the following. First, we need the following lemma:

Lemma (Donsker-Varadhan) For two probability measures $P,Q$ defined on $\mathcal{X}$, define the set of all functions $f : \mathcal{X} \to \mathbb{R}$ with the property $$ \mathbb{E}_Q\left[e^{f(X)}\right]<\infty $$ as $\mathcal{S}$. If $D(P||Q) < \infty$, then for every $f\in\mathcal{C}$, $\mathbb{E}_P[f(X)]$ exists and $$ D(P||Q) = \sup_{f\in\mathcal{C}}\left(\mathbb{E}_p[f(X)] - \log \mathbb{E}_Q\left[e^{f(X)}\right]\right). $$

Proof of Lemma

First, for the function $f = \log\frac{dP}{dQ}$, where $\frac{dP}{dQ}$ is the Radon-Nikodym derivative of $P$ with respect to $Q$, as from the existence of $D(P||Q)$ we know that $P$ is absolutely continuous with respect to $Q$. With this choice of $f$, we get $$ \mathbb{E}_P[f(X)] - \log\mathbb{E}_Q\left[e^{f(X)}\right] = \mathbb{E}_P\left[\log\frac{dP}{dQ}\right] - \log\underbrace{\mathbb{E}_Q\left[\frac{dP}{dQ}\right]}_{=1} = D(P||Q). $$

For the other direction, fix $f \in\mathcal{C}$, and consider the tilted measure $$ \widetilde{Q} = \frac{e^{f(x)}Q(dx)}{\int_{\mathcal{X}}e^{f(x)}Q(dx)}. $$

Note that, $$ \frac{d\widetilde{Q}}{dQ}=\frac{e^{f(x)}}{\int_{\mathcal{X}}e^{f(x)}Q(dx)} \implies \log \frac{d\widetilde{Q}}{dQ} = f(x) - \log\int_{\mathcal{X}}f(x)Q(dx) = f(x) - \log\mathbb{E}_Q[f(X)]. $$

Now, compute the quantity inside supremum via: $$ \mathbb{E}_P[f(X)]-\log\mathbb{E}_Q\left[e^{f(X)}\right]=\mathbb{E}_P\left[\log\frac{d\widetilde{Q}}{dQ}\right]=\mathbb{E}_P\left[\log\frac{d\widetilde{Q}dP}{dQdP}\right] = D(P||Q) - D(P||\widetilde{Q}) \leq D(P||Q). $$

Hence, we are done.

With this, we will now prove the weak lower semi-continuity of divergence. We know that set of all bounded, continuous functions are dense in $L^1$ space. Hence, we can restrict our attention to $\mathcal{C}_b$, namely, the set of all bounded, continuous functions when taking supremum. Now,

\begin{align*} \liminf_{n\to\infty}D(P_n||Q_n) & = \sup_{n\geq 1}\inf_{k\geq n}\sup_{f\in \mathcal{C}_b}\mathbb{E}_{P_n}[f(X)]-\log\mathbb{E}_{Q_n}\left[e^{f(X)}\right] \\ & \geq \sup_{n\geq 1}\sup_{f\in \mathcal{C}_b}\inf_{k\geq n}\mathbb{E}_{P_n}[f(X)]-\log\mathbb{E}_{Q_n}\left[e^{f(X)}\right] \\ & = \sup_{f\in \mathcal{C}_b}\sup_{n\geq 1}\inf_{k\geq n}\mathbb{E}_{P_n}[f(X)]-\log\mathbb{E}_{Q_n}\left[e^{f(X)}\right] \\ & = \sup_{f\in\mathcal{C}_b}\liminf_{n\to\infty}\mathbb{E}_{P_n}[f(X)]-\log\mathbb{E}_{Q_n}\left[e^{f(X)}\right] \\ & = \sup_{f\in\mathcal{C}_b} \mathbb{E}_{P}[f(X)]-\log\mathbb{E}_{Q}\left[e^{f(X)}\right] \\ & = D(P||Q), \end{align*} where the first line is definition of liminf, second line follows from minimax inequality, third line follows as I can swap orders of two supremums, fifth line is a consequence of the equivalent characterization of weak convergence (apparently aka Portmanteau theorem), and the last line is just characterization of KL divergence via Donsker-Varadhan lemma above.

  • $\begingroup$ Thanks! Simple but nice observation. Do you have a proof for the remark? $\endgroup$ – karakusc Apr 24 '17 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.