# Difference between product distribution and joint distribution?

What is the difference between an $$n$$-fold product distribution and a joint distribution with $$n$$ random variables? Is it only defined for independent random variables? I am confused as to what is the definition of a product distribution.

Context: I am reading class notes by John Duchi that say the $$KL$$-Divergence of product distributions $$P = P_1 \times P_2 \ldots P_n$$ and $$Q = Q_1 \times Q_2 \ldots Q_n$$ given by $$KL(P || Q)$$ satisfies the decoupling equality of being $$\sum_{i = 1}^{n}KL(P_i||Q_i)$$.

• They are the same thing when the random variables are independent (See M.P's answer below). Do you follow M.P.'s answer? Mar 8, 2019 at 16:28
• I don't completely understand the definition of product distrib. when the r.v's are not independent. Is there a definition that does not use measure-theoretic notation as much? Or is there only a definition of 'product measure' and not for 'product distribution'? Is it literally the product of probabilities over different supports in the non-independent setting? Are the supports disjoint? Can $Q_1 \ldots Q_n$ in the KL notes be considered as n distributions on n r.v's? Mar 8, 2019 at 18:41

If $$X_1,\ldots,X_n$$ are real-valued random variables on some $$(\Omega,\mathcal F, P)$$, then their joint distribution $$Q_{joint}$$ is the push-forward measure of the $$n$$-dimensional random variable $$(X_1,\ldots,X_n)$$, i.e. $$Q_{joint} \colon \mathcal B (\Bbb R^n)\to [0,1], \quad B \mapsto P\big( (X_1,\ldots,X_n) \in B\big).$$ The product measure $$Q_{prod}$$ is the unique probability measure $$Q_{prod} \colon \mathcal B (\Bbb R^n)\to [0,1]$$ with $$Q_{prod}(B_1\times\ldots\times B_n)= P( X_1\in B_1)\cdots P(X_n\in B_n)$$ for all $$B_1,\ldots,B_n\in\mathcal B (\Bbb R)$$.

Both $$Q_{joint}$$ and $$Q_{prod}$$ are defined regardless of independence. However, the random variables $$X_1,\ldots,X_n$$ are independent under $$P$$ if and only if $$Q_{joint}=Q_{prod}$$.

The same extends to random variables taking values in spaces other than $$\Bbb R$$.

Here is an exmaple that does not mention measures.

Suppose that $$X\in\{0,1\}$$ and $$Y\in\{0,1\}$$ are integer valued random variables with joint distribution P. Further suppose that $$P_X(X=1)=0.8$$, $$P_X(X=0)=0.2$$, $$P_Y(Y=1)=0.7$$, $$P_Y(Y=0)=0.3$$.

If $$X$$ and $$Y$$ are independent, then their joint distribution and the product distribution $$P=P_X\times P_Y$$ are the same --- $$P(X=1 \ \mathrm{and}\ Y=1)=0.56$$, $$P(X=1 \ \mathrm{and}\ Y=0)=0.24$$, $$P(X=0 \ \mathrm{and}\ Y=1)=0.14$$, and $$P(X=0 \ \mathrm{and}\ Y=0)=0.06$$.

If they are not independent, then their joint distribution $$P_{XY}$$ could be any distribution with the same marginals. For example, their joint distribution could be $$P_{XY}(X=1 \ \mathrm{and}\ Y=1)=0.7$$, $$P_{XY}(X=1 \ \mathrm{and}\ Y=0)=0.1$$, $$P_{XY}(X=0 \ \mathrm{and}\ Y=1)=0.0$$, and $$P_{XY}(X=0 \ \mathrm{and}\ Y=0)=0.2$$.

• but what is the prod. distrib in this example in non-independent case? You only defined joint for this case where I do see that marginals match up. as in $0.7+0.1 = P_X(X=1)$. Mar 8, 2019 at 19:28
• Product distribution of $P_X$ and $P_Y$ is the $P$ where 𝑃(1,1)=.56, P(1.0)=.24, P(0,1)=0.14, and P(0,0)=0.06. This is true even if $X$ and $Y$ are not independent. Mar 8, 2019 at 23:01
• The phrase "Product Distribution" is also used for different concept. (See, en.wikipedia.org/wiki/Product_distribution .) Mar 8, 2019 at 23:04