# Linearity of expectations - Why does it hold intuitively even when the r.v.s are correlated?

An experiment - say rolling a die, is performed a large number of times, $n$. Let $X$ and $Y$ be two random variables that summarize this experiment.

Intuitively(by the law of large numbers), if I observe the values of $X$, over a large number of trials, take their mean, $m_{X}=\frac{1}{n}\sum_{i}{x_{i}}$, and observe the values of $Y$, take their mean $m_{Y}=\frac{1}{n}\sum_{i}{y_{i}}$ and the add the two column means, this is very close to $E(X)+E(Y)$.

If we observe the values of $X+Y$ in a third column, and take their arithmetic mean, $m_{X+Y}$, this will be very close to $E(X+Y)$.

Therefore, linearity of expectation, that $E(X+Y)=E(X)+E(Y)$ emerges as a simple fact of arithmetic (we're just adding two numbers in different orders).

I know linearity of expectations holds, even when the $X$ and $Y$ are dependent. For example, the binomial and hypergeometric expectation is $E(X)=np$, although in the binomial story, the $Bern(p)$ random variables are i.i.d., but in the hypergeometric story, they are dependent.

If two random variables are correlated, wouldn't that affect the average of their sum, than if they were uncorrelated? Any insight or intuition would be great!

• Just an extreme case for intuition. If $X=Y$ then $E(X+Y)=E(2X)=2E(X)$.
– user223391
Commented Jan 2, 2017 at 4:00
• I would observe that I like your intuitive argument about sample means and note that you make no assumption of independence. Commented Jan 2, 2017 at 4:07

For intuition, suppose the sample space consists of a finite number of equally probable outcomes (this is of course not true for all probability spaces, but many situations can be approximated by something of this form). Then $$E(X+Y) = \frac{(x_1+y_1)+(x_2+y_2)+\cdots+(x_n+y_n)}n$$ and $$E(X)+E(Y) = \frac{x_1+x_2+\cdots+x_n}n + \frac{y_1+y_2+\cdots+y_n}n$$ which is obviously the same.

Say you have $X$ and $Y$ independent and then you turn up the correlation. Say they're mean zero too, just for simplicity. Then $X+Y$ will still be positive just as often on average as it's negative. It's just that it will be more likely that X and Y are positive or negative together. Thus the mean of $X+Y$ stays zero. However, it does increase the variance since $X+Y$ will tend to be larger in magnitude cause $X$ and $Y$ have the same sign more often.

If two random variables are correlated, wouldn't that affect the average of their sum, than if they were uncorrelated?

Being correlated or uncorrelated matters when we have $\mathsf{E}(XY)$ terms. So for uncorrelated RVs, $\mathsf{E}(XY)=\mathsf{E}(X)\mathsf{E}(Y)$. Obviously, when we consider expectation of sum of RVs, or sum of expectations, no such terms appear. Hence being correlated or not does not change anything.

• You are wrongly extending this answer to some nonlinear example. Your impression is strange since this post is about linearity of expectation.
– msm
Commented Jan 2, 2017 at 9:38
• I'm not extending this answer to anything. I'm merely pointing out that as written the reasoning is logically unsound. You state correctly that expectation does not commute with multiplication in general, but then you say that no products appear in a sum of RVs. That is completely irrelevant to whether expectation commutes with addition. So your post simply fails to explain why expectation is linear. Commented Jan 2, 2017 at 11:04
• Furthermore, you deleted your very first comment in response to mine, in which you stated falsely that correlation has no effect on the two expressions I gave. When I corrected you, you did not acknowledge your error but instead said that I'm making a mistake. Trust me, I have a solid grasp of logic and it's not mine that is flawed. If I make an error prove me wrong rigorously and I'll acknowledge it. Commented Jan 2, 2017 at 11:07
• First: regarding "your post simply fails to explain why expectation is linear": why should it explain it? The question is something else. Second: "you deleted your very first comment in response to mine" that was because of your mistake in your comment when you "mistyped". Third: I didn't say your argument is wrong or question your grasp of logic. Your example was just irrelevant. Also notice that I don't tend to prove anyone's error on this website as much as I did not see any relevant proof of my answer being wrong. Besides, you already downvoted! So there is no point for further argument.
– msm
Commented Jan 2, 2017 at 11:40
• The question asks for intuitive explanation. Absolute rigour is not needed, but your post essentially comes down to the common fallacy that if you have $A \to B$ then you have $\neg A \to \neg B$. That is the main point of my disagreement. My example is relevant because your post says expectation does not distribute over product, and then erroneously implies that if there is no product term then correlation has no effect. I disproved that using a different operation. Here is another example: $E(|X+Y|) \ne E(|X|) + E(|Y|)$ in general but equal if $X = Y$. I make my points for other readers. Commented Jan 2, 2017 at 13:23

Here's my non-rigorous attempt:

We know by the law of total expectation that $$E[X] = E[E[X|Y]]$$

or in a special case, intuitively if $A_i$s partition the sample space $$E[X] = \sum_i E[X|A_i]P(A_i)$$

That means even when $X$ is dependent on $Y$, $E[X]$ already knows about and has accounted for this dependency! How $X$ and $Y$ affect each other were known to $E[X]$ and $E[Y]$.

I will give a reasoning considering the dependence between $$X$$ and $$Y$$. \begin{align} E(X+Y) & = \sum_y\sum_x(x+y)p(x,y) \\& =\sum_y\sum_x(x+y)p(x)p(y|x) \\ &= \sum_y\sum_xxp(x)p(y|x) + \sum_y\sum_xyp(x)p(y|x) \\&=\sum_xxp(x)\sum_yp(y|x) + \sum_xp(x)\sum_yyp(y|x) \\&=\sum_xxp(x) + \sum_xp(x)E(Y|X=x) \\&=E(X) + E_X(E(Y|X)) \\&=E(X) + E(Y) \end{align} So $$E(X+Y)$$ can be transformed into any of the following three: $$E(X + E(Y|X))$$, $$E(X) + E_X(E(Y|X))$$, $$E(X)+E(Y)$$. The first one is more intuitive, but the last one is more simple.