0
$\begingroup$

The mutual information between random variables $X$ and $Y$ is defined (in terms of their joint PDF or PMF $p$) [Wikipedia]

$$ I(X; Y) = \mathbb{E}_{p(X, Y)}\left[ \log \frac{p(X, Y)}{p(X)p(Y)} \right]. $$

Equivalently we can write $$ I(X; Y) = \mathbb{E}_{p(X, Y)}\left[ \log {p(X, Y)} \right] - \mathbb{E}_{p(X, Y)}\left[\log {p(X)p(Y)} \right]. $$

When doing some calculations, I got the following quantity, which is the same but swapping the independent and joint distributions in the right-hand side. Is there a name for it, or an intuitive interpretation of it?

$$ F(X, Y) = \mathbb{E}_{p(X, Y)} \left[ \log p(X, Y) \right] - \mathbb{E}_{p(X)p(Y)} \left[ \log p(X, Y) \right] $$

$\endgroup$
5
  • $\begingroup$ What does $\mathbb{E}_{p(X)p(Y)}$ mean? I know that $\mathbb{E}_{p(X,Y)}$ means the expected value. $\endgroup$
    – Joe
    Commented Oct 22, 2019 at 11:33
  • $\begingroup$ The expected value, but over the marginal distributions of X and Y instead of over their joint. $$\mathbb{E}_{p(X)p(Y)}[f(X, Y)] = \mathbb{E}_{p(X)} \mathbb{E}_{p(Y)} f(X, Y) $$ $\endgroup$
    – rhaps0dy
    Commented Oct 22, 2019 at 14:47
  • $\begingroup$ If you compute $\mathbb{E}_{p(Y)}f(X,Y)$ as a function of $X$, then compute the expected value of that function over the distribution of $X$, doesn’t it give you $\mathbb{E}_{p(X,Y)}f(X,Y)$? $\endgroup$
    – Joe
    Commented Oct 22, 2019 at 15:37
  • $\begingroup$ No, it does not. That would only be true if you first computed $\mathbb{E}_{p(Y | X)}f(X, Y)$ as a function of $X$. $\endgroup$
    – rhaps0dy
    Commented Oct 23, 2019 at 10:05
  • $\begingroup$ You should ask this on Cross Validated. You’ll probably get more responses. $\endgroup$
    – Joe
    Commented Oct 23, 2019 at 19:21

1 Answer 1

1
$\begingroup$

Mutual information can be written as the Kullback-Leibler (KL) divergence between the joint distribution $p(X,Y)$ and the product distribution $p(X)p(Y)$, $$I(X;Y) = D_{KL}(p(X,Y)\Vert p(X)p(Y))$$

There is also a symmetrized version of KL divergence, sometimes called Jeffreys divergence or J-divergence, $D_J(p\Vert q) = D_{KL}(p\Vert q) + D_{KL}(q\Vert p)$ (see e.g. https://arxiv.org/pdf/1009.4004.pdf).

Your expression $F(X,Y)$ is the Jeffreys divergence between the joint distribution $p(X,Y)$ and product distribution $p(X)p(Y)$: \begin{align} D_J(p(X,Y)\Vert p(X)p(Y)) &= I(X;Y) + D_{KL}(p(X)p(Y)\Vert p(X,Y))\\ & = H(X) + H(Y) - H(X,Y)- H(X)-H(Y) - \mathbb{E}_{p(X)p(Y)}[\log p(X,Y)] \\ & = \mathbb{E}_{p(X,Y)}[\log p(X,Y)] - \mathbb{E}_{p(X)p(Y)}[\log p(X,Y)] \end{align}

The "reverse" KL term $D_{KL}(p(X)p(Y)\Vert p(X,Y))$ is sometimes called lautum information, and is described in the following:

  • Palomar and Verdú, "Lautum Information", IEEE Trans on Info Theory, 2008.
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .