1
$\begingroup$

Let $Q_1, \dots, Q_n$ be arbitrary distributions over the finite sets $X_1, \dots, X_n$, and $P$ be an arbitrary distribution over $X_1 \times \cdots \times X_n$ with marginals $P_1, \dots, P_n$.

How can I prove that $$D(P||Q_1\times \cdots \times Q_n)=D(P||P_1\times \cdots \times P_n) + \sum_{i=1}^n D(P_i||Q_i),$$ and conclude that among the distributions $P$ with marginals $P_1, \dots, P_n$ the I-divergence $D(P||Q_1\times \cdots \times Q_n)$ is minimal if $P = P_1\times \cdots \times P_n$?

$\endgroup$

1 Answer 1

1
$\begingroup$

\begin{align} D(P\,||\,Q_1 \times ... \times Q_n) &= \sum\limits_{x_1, ..., x_n} p(x_1, ..., x_n) \log\left(\frac{p(x_1, ..., x_n)}{q(x_1) \times ... \times q(x_n)}\right) \\ &= \sum\limits_{x_1, ..., x_n} p(x_1, ..., x_n) \log\left(\frac{p(x_1)\times ... \times p(x_n)}{q(x_1) \times ... \times q(x_n)}\right) \\ &\quad + \sum\limits_{x_1, ..., x_n} p(x_1, ..., x_n) \log\left(\frac{p(x_1, ..., x_n)}{p(x_1) \times ... \times p(x_n)}\right) \\ &= \sum\limits_{i=1}^n \sum\limits_{x_i}p_i(x_i) \log\left(\frac{p(x_i)}{q(x_i)}\right) + D(P\,||\,P_1 \times ... \times P_n) \\ &= \sum\limits_{i=1}^n D(P_i\,||\,Q_i) + D(P\,||\,P_1 \times ... \times P_n). \end{align}

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .