Existence of independent and identically distributed random variables.

I often see the sentence "let $X_1, X_2, \ldots$ be a sequence of i.i.d. random variables with a certain distribution". But given a random variable $X$ on a probability space $\Omega$, how do I know that there is a sequence of INDEPENDENT random variables of the same distribution on $\Omega$?

• This is Kolmogorov's consistency theorem. See for example Varadhan's Probability Theory page 41 for a nice explanation. It's available on Google books. Nov 28 '12 at 12:29
• @Stefan Hansen : To apply Kolmogorov's consistency theorem, you need the law to be a probability measure on a Polish space. Nevertheless, this result holds in full generality, and can be seen as a particular case of Ionescu-Tulcea's theorem. Nov 28 '12 at 12:56
• I have also to mention than the sequence is defined on a new probability space, and not on the original space $\Omega$ where $X$ is defined. This is not a real issue, because of the philosophy of probability theory : this is the laws of random variables which are of interest, rather than the probability space on which they are defined. Nov 28 '12 at 12:59

The easiest way to show the existence in our case is to construct such a probability space. The intuition is that it is easy to define the joint probability measure on some "simple" sets using the iid property and then one leverages the celebrated Caratheodory extension theorem.

More precisely, let $(E,\mathcal E)$ be the measurable space where our random variables are supposed to take values and let $\mu$ be a probability measure on $(E,\mathcal E)$ representing the distribution of such random variables. Define $\Omega = E^{\mathbb N_0}$ to be the space of countable trajctories over $E$ and let $\mathcal F$ be its product $\sigma$-algebra. Define the probability measure $\mathsf P$ on $(\Omega,\mathcal F)$ just based on the independence, i.e. for any $A_0,\dots,A_N\in \mathcal E$ we put $$\mathsf P(X_0\in A_0,\dots,X_n\in A_n):=\mu(A_0)\times \dots\times \mu(A_n).$$ So far the measure $\mathsf P$ is only defined on the collection $\mathcal A$ of measurable rectangles, i.e. subset of $\Omega$ of the form $A_0\times\dots\times A_n\times \Omega\times\Omega\times\dots$ where $A_i\in \mathcal E$. Finite unions of elements of $\mathcal A$ form the algebra, say $\mathcal B$. Clearly, $\mathsf P$ is a finite pre-measure on $\mathcal B$ and hence by the Caratheodory extension theorem we obtain the unique measure $\mathsf P$ on $\mathcal F = \sigma(\mathcal B)$.

As Ahriman has pointed out, if you are given a random variable $X:\Omega\to E$ it may not be possible to construct the whole sequence on $\Omega$ as the latter may be quite a poor space, so you would have to go for a richer space. For example, $E$ always can be considered as a sample space for the distribution over it, by taking $\mathrm id_E$ being a random variable. But in case $E = \{a,b\}$ and you have $\mu(a) = 0.4$ and $\mu(b) = 0.6$ it is only possible to construct one and only one random variable defined on $E$ which has $\mu$ as a distribution.

• +1. Nice answer. Nov 28 '12 at 13:13

A very concrete approach to construct a sequence of iid random variables with distribution $$F$$ (say, $$F$$ is the desired cumulative distribution function) is to proceed as follows.

It is enough to construct a sequence of iid uniform $$[0,1]$$ random variables $$(X_i)_{i\ge 1}$$, because then $$(F^{\leftarrow}(X_i) )_{i\ge 1}$$ is a sequence of iid random variables with common distribution $$F$$. Here, $$F^\leftarrow$$ is the generalized inverse of $$F$$.

So let's concretely construct a sequence of iid uniform $$[0,1]$$ random variable. Take $$\Omega = [0,1]$$ and $$\mathbf P$$ be the Lebesgue measure on $$\Omega$$; there we naturally have a single measurable function (or random variable) $$U:\Omega\to\mathbb R$$ defined by $$U(x)=x$$ for any $$x\in \Omega$$.

Now view $$U$$ as a random variable, and write its decomposition in base $$2$$ as $$U = \sum_{k=1}^{+\infty} B_k/2^k.$$ Exercise: show that each $$(B_k)_{k\ge 1}$$ is a sequence of iid Bernoulli$$(1/2)$$ random variables on $$\Omega$$.

Then we may split the set of positive integers into countably many infinite subsets; for instance, let $$(p_n)_{n\ge 1}$$ be the sequence of prime numbers and fr any $$n\ge 1$$, let $$I_n = \{ p_n^j, j=1,2,3,4,... \}$$. Each $$I_n$$ is infinite and $$I_n\cap I_m = \emptyset$$ for any $$n\ne m$$. From these sets $$I_n$$, for any $$n$$ define $$X_n = \sum_{k=1}^\infty B_{\varphi_n(k)}/2^k$$ where $$\varphi_n$$ is a bijection $$\mathbb N_{\ge 1}\to I_n$$ (that is, the countable set $$I_n$$ can be written as $$I_n = \{\varphi(1), \varphi(2),\varphi(3), ...\}$$. Because the sets $$I_n$$ are disjoint, a Bernoulli random variable $$B_k$$ is used only in one of the $$X_n$$'s, so we can prove the following to conclude:

Exercise 2: show that each $$X_n$$ has the uniform distribution on $$[0,1]$$, and that the sequence $$(X_1,...,x_n,...,)$$ is iid.

Hence $$\Omega=[0,1]$$ equipped with the Lebesgue measure is rich enough to construct a sequence of iid random variables with distribution $$F$$.

Since they're i.i.d., you can just use a product measure on a product space $\Omega\times\Omega\times\Omega\times\cdots$.

And for many purposes, you can take $\Omega=\mathbb R$ and let the measurable subsets of $\Omega$ be the Borel sets.

• @cantorhead : I wasn't suggesting you wouldn't need a product of measure spaces; rather I was saying you could take $\Omega = \mathbb R$ and use the product of copies of $\Omega.$ But here was the question: "But given a random variable $X$ on a probability space $\Omega$, how do I know that there is a sequence of INDEPENDENT random variables of the same distribution on $\Omega.$?" If $X_1$ has $\Omega$ as its domain, then in some cases there is no infinite i.i.d. sequence, and in some cases no finite i.i.d. sequence of length more than $1$, on the same domain $\Omega$ and$\,\ldots\qquad$ Dec 17 '16 at 20:22
• $\ldots\,$ with each member of the sequence having that same probability distribution. It's easy to find counterexamples with some discrete distributions. But the product of countably many copies of $\Omega$ does it. $\qquad$ Dec 17 '16 at 20:23
• @cantorhead : You said "I often see the sentence 'let $X_1, X_2, \ldots$ be a sequence of i.i.d. random variables with a certain distribution' ". Nothing in that presupposes that if $X_1$ has a certain probability space as its domain, then there is such an i.i.d. sequence on the SAME domain. It only presupposes that there is SOME domain on which such an i.i.d. sequence exists. Dec 17 '16 at 21:51
• Dec 17 '16 at 21:59
• I've moved my comments to a new question: math.stackexchange.com/q/2063785/118481. Thank you for the input, it helped in making the new question more precise. Dec 18 '16 at 19:29