Can someone define independence of two random variables with this "product rule", or are there any counterexamples?
-
3$\begingroup$ Sure there are, try X standard normal and Y=UX with U symmetric Bernoulli. $\endgroup$– DidOct 29, 2015 at 19:13
-
$\begingroup$ @Did, what do you mean by symmetric Bernoulli? $P(U=0) = P(U=1)$? $\endgroup$– Carlos H. Mendoza-CardenasOct 29, 2015 at 19:16
-
$\begingroup$ P(U=1)=P(U=-1)=1/2 $\endgroup$– DidOct 29, 2015 at 19:29
-
$\begingroup$ @Did Is that a Bernoulli variable? I don't think so. $\endgroup$– Carlos H. Mendoza-CardenasOct 29, 2015 at 19:32
-
2$\begingroup$ @CarlosMendoza Many people call Bernoulli what you probably call a two-points distribution. $\endgroup$– DidOct 29, 2015 at 19:33
1 Answer
Independence implies uncorrelation
We say that two random variables are uncorrelated if $E[XY] = E[X]E[Y]$.
If $X$ and $Y$ are independent and discrete (this can also be extended to continuous random variables), then
$$p_{X,Y}(x,y) = p_X(x)p_Y(y)\qquad \forall (x,y)$$
Using this in the definition of $E[XY]$ and doing some rearrangements of terms, we have
\begin{align} E[XY] &= \sum_x\sum_y xyp_{X,Y}(x,y)\\ &= \sum_x\sum_y xyp_X(x)p_Y(y)\\ &= \sum_x\sum_y [xp_X(x)][yp_Y(y)]\\ &= \sum_xxp_X(x)\sum_yyp_Y(y)\\ &= E[X]E[Y] \end{align}
Therefore, if $X$ and $Y$ are independent, then they are uncorrelated.
$\\$
Uncorrelation does not imply independence (in general)
In general$^1$, it is false to say that if $X$ and $Y$ are uncorrelated, then they are independent. Let's see a counterexample. Let $X$ and $Y$ be random variables taking values in $\{(0,1), (1,0), (-1,0), (0,-1)\}$ with equal probability $0.25$. Then, $XY = 0$ with probability $1$, and therefore $E[XY] = 0$.
By other hand,
$\displaystyle p_X(x) = \left\{ \begin{array}{ll} \frac{1}{4} & x=-1\\ \frac{1}{2} & x=0\\ \frac{1}{4} & x=1\\ \end{array} \right. $ $\hspace{1cm}$ and $\hspace{1cm}$ $p_Y(y) = \left\{ \begin{array}{ll} \frac{1}{4} & y=-1\\ \frac{1}{2} & y=0\\ \frac{1}{4} & y=1\\ \end{array} \right. $
which implies that $E[X]=E[Y]=0$. Then, $E[XY] = E[X]E[Y]$ and the random variables are uncorrelated.
However,
$$p_X(1)p_Y(1) = \frac{1}{4}\frac{1}{4} \neq p_{X,Y}(1,1) = 0$$
which shows that $X$ and $Y$ are not independent.
We can conclude then that independence is a sufficient but not a necessary condition for uncorrelation.
$^1$ The only two exceptions I am aware of are the jointly Gaussian variables and two Bernoulli random variables...any other?
-
$\begingroup$ So you group terms into 2 sums because y's terms are not dependent on x's sum and vice versa please? $\endgroup$– AvvApr 6, 2022 at 22:33
-
1$\begingroup$ @Avv. I edited my answer to add two additional steps and hopefully make it clearer. In the third line, you move the factor $xp_X(x)$ out of the sum $\sum_y$ because it doesn't depend on $y$, only on $x$. $\endgroup$ Apr 7, 2022 at 13:40
-
$\begingroup$ I upvoted the answer. It's very clear now. Thank you. $\endgroup$– AvvApr 7, 2022 at 18:57
-
$\begingroup$ One more thing please. Why if $g(y) = E[X|Y=y]$ then $E[E[X|y=y]]$ is not expectation of a number but expectation of a random variable? Should not $E[X|y=y]$ be treated as a number not a random variable please so we have $E[E[X|y=y]] = E[X|Y=y]$? $\endgroup$– AvvApr 7, 2022 at 19:47
-
$\begingroup$ @Avv, I am not sure how your question relates to the original question and my answer. Please elaborate. Your question seems to be a new question, and you should probably ask it in a new post, but I will try to answer in the next comment.... $\endgroup$ Apr 8, 2022 at 17:34