# Trying to understand the behaviour of i.i.d.

In a course called introduction to probability theorem we are covering now i.i.d. (independent and identically distributed random variables). I already know when two variables are independent: $X, Y$ are independent if $P(X \le a, Y \le b) = P(X \le a) \cdot P(Y \le b)$ for all possible choices of $a, b$.

The following definition comes from Wikipedia:

In probability theory and statistics, a sequence or other collection of random variables is independent and identically distributed (i.i.d.) if each random variable has the same probability distribution as the others and all are mutually independent.

What does "the same probability distribution" mean? That $P(X \le a) = P(y \le a)$ (assuming that we are interested in only two random variables, $X$ and $Y$)? What are (not necessary interesting) examples of i.i.d. variables, that aren't discrete?

This exercise has given rise to serious doubts regarding the understanding of random variables:

14. Let $X, Y$ have the same probability distribution. Is it true that $\mathbb E[\frac{X}{X+Y}] = \mathbb E[\frac{Y}{X+Y}]$?

At first I thought that the answer if affirmative, but I've heard that there is a counterexample (which I can't find). If it's negative, what is wrong with my intuition behind such equalities?

• Counterexample: assume that (X,Y) is uniform on the set {(0,1),(1,2),(2,0)} then X and Y are both uniform on {0,1,2} but E(X/(X+Y))=4/9 while E(Y/(X+Y))=5/9. – Did Apr 20 '15 at 17:37

Very good questions.

If $X$ and $Y$ have the same probability distribution, than for every $z$, $$\Pr\left(X \leqslant z\right) = \Pr\left(Y \leqslant z\right)$$ Then $X$ and $Y$ are said to be identically distributed. Note that this does not imply independence. A classic example would be binormally distributed vector of $(X,Y) \sim \mathcal{BN}\left(\rho\right)$: $$f_{X,Y}\left(x,y\right) = \frac{1}{2\pi} \frac{1}{\sqrt{1-\rho^2}} \exp\left(-\frac{1}{2} \frac{x^2+y^2 + 2 \rho x y}{1-\rho^2}\right)$$ It is straightforward to show that $f_X(z) = f_Y(z) = \frac{1}{2\pi}\exp\left(-\frac{1}{2} z^2\right)$, yet $X$ and $Y$ are not independent, in fact $$\mathbb{E}\left(X Y\right) = \rho \not= \mathbb{E}\left(X\right) \mathbb{E}\left(Y\right) = 0$$ unless $\rho = 0$.

Regarding your second question, equality $\mathbb{E}\left(\frac{X}{X+Y}\right) = \mathbb{E}\left(\frac{Y}{X+Y}\right)$ requires exchangeability, i.e. $f_{X,Y}\left(x,y\right) = f_{X,Y}\left(y,x\right)$ for all $x$ and $y$, which does not follow from the identical distributions of $X$ and $Y$.

I used Mathematica to build a counter-example. Let $(X,Y)$ be a pair of discrete random variables from 1 to 3 with the following probability mass function:

$$p_{X,Y}\left(x,y\right) = \begin{cases} \frac{1}{17} & (x=2\land y=2)\lor (x=2\land y=1) \\ \frac{14}{51} & (x=2\land y=3)\lor (x=3\land y=1) \\ \frac{1}{3} & x=1\land y=2 \end{cases}$$

Here is how I did it (code for this can be retrieved from pastebin).

First define a custom distribution over tuples of integers from 1 to 3:

p = EmpiricalDistribution[Array[pr, 3^2] -> Tuples[Range[3], 2]];


Now request that marginal CDFs are the same, and yet expectations are different:

In[58]:= {inst} =
FindInstance[
sol = Reduce[
ForAll[x,
CDF[MarginalDistribution[dist, 1], x] ==
CDF[MarginalDistribution[dist, 2], x]] &&
DistributionParameterAssumptions[dist] &&
Array[0 <= pr[#] <= 1 &, 3^2, 1, And] &&
Expectation[x/(x + y), {x, y} \[Distributed] dist] !=
Expectation[y/(x + y), {x, y} \[Distributed] dist], Reals],
Array[pr, 3^2]]

Out[58]= {{pr[1] -> 0, pr[2] -> 1, pr[3] -> 0, pr[4] -> 3/17,
pr[5] -> 3/17, pr[6] -> 14/17, pr[7] -> 14/17, pr[8] -> 0,
pr[9] -> 0}}


Show the probability mass function:

In[62]:= PiecewiseExpand[PDF[counterExampleDist = dist /. inst, {x, y}]]

Out[62]= Piecewise[{{1/
17, (x == 2 && y == 2) || (x == 2 && y == 1)}, {14/
51, (x == 2 && y == 3) || (x == 3 && y == 1)}, {1/3,
x == 1 && y == 2}}, 0]


Now verify:

In[63]:= {CDF[MarginalDistribution[counterExampleDist, 1], x],
CDF[MarginalDistribution[counterExampleDist, 2], x]}

Out[63]= {1/3 Boole[1 <= x] + 20/51 Boole[2 <= x] +
14/51 Boole[3 <= x],
1/3 Boole[1 <= x] + 20/51 Boole[2 <= x] + 14/51 Boole[3 <= x]}

In[64]:= {Expectation[x/(
x + y), {x, y} \[Distributed] counterExampleDist],
Expectation[y/(x + y), {x, y} \[Distributed] counterExampleDist]}

Out[64]= {379/765, 386/765}