4
$\begingroup$

What is the difference between an $n$-fold product distribution and a joint distribution with $n$ random variables? Is it only defined for independent random variables? I am confused as to what is the definition of a product distribution.

Context: I am reading class notes by John Duchi that say the $KL$-Divergence of product distributions $P = P_1 \times P_2 \ldots P_n$ and $Q = Q_1 \times Q_2 \ldots Q_n$ given by $KL(P || Q)$ satisfies the decoupling equality of being $\sum_{i = 1}^{n}KL(P_i||Q_i)$.

$\endgroup$
2
  • 1
    $\begingroup$ They are the same thing when the random variables are independent (See M.P's answer below). Do you follow M.P.'s answer? $\endgroup$
    – irchans
    Mar 8, 2019 at 16:28
  • $\begingroup$ I don't completely understand the definition of product distrib. when the r.v's are not independent. Is there a definition that does not use measure-theoretic notation as much? Or is there only a definition of 'product measure' and not for 'product distribution'? Is it literally the product of probabilities over different supports in the non-independent setting? Are the supports disjoint? Can $Q_1 \ldots Q_n$ in the KL notes be considered as n distributions on n r.v's? $\endgroup$
    – hearse
    Mar 8, 2019 at 18:41

2 Answers 2

3
$\begingroup$

If $X_1,\ldots,X_n$ are real-valued random variables on some $(\Omega,\mathcal F, P)$, then their joint distribution $Q_{joint}$ is the push-forward measure of the $n$-dimensional random variable $(X_1,\ldots,X_n)$, i.e. $$ Q_{joint} \colon \mathcal B (\Bbb R^n)\to [0,1], \quad B \mapsto P\big( (X_1,\ldots,X_n) \in B\big). $$ The product measure $Q_{prod}$ is the unique probability measure $$ Q_{prod} \colon \mathcal B (\Bbb R^n)\to [0,1] $$ with $$ Q_{prod}(B_1\times\ldots\times B_n)= P( X_1\in B_1)\cdots P(X_n\in B_n)$$ for all $B_1,\ldots,B_n\in\mathcal B (\Bbb R)$.

Both $Q_{joint}$ and $Q_{prod}$ are defined regardless of independence. However, the random variables $X_1,\ldots,X_n$ are independent under $P$ if and only if $Q_{joint}=Q_{prod}$.

The same extends to random variables taking values in spaces other than $\Bbb R$.

$\endgroup$
2
$\begingroup$

Here is an exmaple that does not mention measures.

Suppose that $X\in\{0,1\}$ and $Y\in\{0,1\}$ are integer valued random variables with joint distribution P. Further suppose that $P_X(X=1)=0.8$, $P_X(X=0)=0.2$, $P_Y(Y=1)=0.7$, $P_Y(Y=0)=0.3$.

If $X$ and $Y$ are independent, then their joint distribution and the product distribution $P=P_X\times P_Y$ are the same --- $P(X=1 \ \mathrm{and}\ Y=1)=0.56$, $P(X=1 \ \mathrm{and}\ Y=0)=0.24$, $P(X=0 \ \mathrm{and}\ Y=1)=0.14$, and $P(X=0 \ \mathrm{and}\ Y=0)=0.06$.

If they are not independent, then their joint distribution $P_{XY}$ could be any distribution with the same marginals. For example, their joint distribution could be $P_{XY}(X=1 \ \mathrm{and}\ Y=1)=0.7$, $P_{XY}(X=1 \ \mathrm{and}\ Y=0)=0.1$, $P_{XY}(X=0 \ \mathrm{and}\ Y=1)=0.0$, and $P_{XY}(X=0 \ \mathrm{and}\ Y=0)=0.2$.

$\endgroup$
3
  • $\begingroup$ but what is the prod. distrib in this example in non-independent case? You only defined joint for this case where I do see that marginals match up. as in $0.7+0.1 = P_X(X=1)$. $\endgroup$
    – hearse
    Mar 8, 2019 at 19:28
  • $\begingroup$ Product distribution of $P_X$ and $P_Y$ is the $P$ where 𝑃(1,1)=.56, P(1.0)=.24, P(0,1)=0.14, and P(0,0)=0.06. This is true even if $X$ and $Y$ are not independent. $\endgroup$
    – irchans
    Mar 8, 2019 at 23:01
  • $\begingroup$ The phrase "Product Distribution" is also used for different concept. (See, en.wikipedia.org/wiki/Product_distribution .) $\endgroup$
    – irchans
    Mar 8, 2019 at 23:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .