Mathematical description of a random sample

Mathematical description of a random sample: which one is it and why?

1. $X_1(\omega), X_2(\omega), ..., X_n(\omega)$, where $X_1, ..., X_n$ are different but i.i.d. random variables.

2. $X(\omega_1), X(\omega_2), ..., X(\omega_n)$, where $X$ is a (single) random variable.

• Option 1.  
– Did
Jun 25, 2011 at 16:45
• What about the "why" part?
– Leo
Jun 25, 2011 at 16:51
• Because the whole purpose of probability theory is to avoid actually manipulating little-omegas. I know that in some first courses one wastes an incredible amount of time specifying what is Omega... This is nonsense, all the quantities one needs are in fact "at the other end of the arrow". The best option is to establish once and for all that suitable spaces Omega exist that are sufficient for the families of random variables one has in mind, and then to proceed happily. .../...
– Did
Jun 25, 2011 at 17:11
• .../... Leo Breiman must have explained this much better, somewhere. (OK, the structure of Omega may become an interesting question again, much later--but not before one begins to tackle questions of a much higher level of sophistication.)
– Did
Jun 25, 2011 at 17:11
• Then, for $X(t_i)$ to be random one needs to pick $t_i$ randomly in the set $\{\omega_1,\ldots,\omega_6\}$ (otherwise $X(t_i)$ is a number and not a random variable). Question: how do you choose $t_i$? The answer seems to be that you choose $t_i$ at random, in other words $t_i$ is a map from an unspecified probability space $S$ to $\{\omega_1,\ldots,\omega_6\}$... in other words, you simply encoded each random variable $X_i$ of option 1. as a random variable $X\circ t_i$ defined on $S$. In the end, option 2. does not exist. (Unrelated: please use the @ sign to notify your comments.)
– Did
Jul 31, 2011 at 14:33

4 Answers

Let's say that the result of an experiment is a n-tuple of real numbers. When we accept 1. as a model of our experiment, we have a probability space $\Omega$ and a random variable $$X: \Omega \to \mathbb{R}^n$$ The outcome of an experiment corresponds to a $\omega \in \Omega$ and therefore to an n-tuple $(X_1(\omega), ..., X_n(\omega))$. This model allows us to ask if the elements of this n-tuple are independent and if not, what their joint distribution is.

If we accept 2. as a model, we have a probability space $\Omega$ and a tuple of random variables $$X_i: \Omega \to \mathbb{R}$$ so that the n-tuple is a random variable of the probability space $\Omega^n$ (Cartesian product). So, in this case, the independence of the elements of the tuple is built into the model. If the elements of the tuple are supposed to be independent, it does not matter.

Note that in the first case we can set $X_i = X_j$, either strict or modulo a null set; in this case we will have a tuple of identically distributed random variables. Choice no.1 does not necessarily imply that the elements of the n-tuple are different random variables (either strictly different or modulo a null set).

Thanks to the stimulating discussion with @Didier, I've clarified something for myself. From the technical standpoint, we have $n$ random variables in the option 1, and $n$ numbers in the option 2. The problem with this may be illustrated by the following example. Consider 3 different people producing their own random samples of size 5 by throwing a die. Here is what they get:

1 person: 1, 3, 1, 4, 2 $\;\;\;\rightarrow$ $X_1(\omega'), X_2(\omega'), ..., X_5(\omega')$
2 person: 2, 2, 1, 6, 3 $\;\;\;\rightarrow$ $X_1(\omega''), X_2(\omega''), ..., X_5(\omega'')$
3 person: 1, 4, 2, 1, 3 $\;\;\;\rightarrow$ $X_1(\omega'''), X_2(\omega'''), ..., X_5(\omega''')$

On the right side, I used option 1 to code these outcomes. How to code them using option 2? We can try this:

1 person: 1, 3, 1, 4, 2 $\;\;\;\rightarrow$ $X(t_1), X(t_2), ..., X(t_5)$
2 person: 2, 2, 1, 6, 3 $\;\;\;\rightarrow$ $X(t_1), X(t_2), ..., X(t_5)$
3 person: 1, 4, 2, 1, 3 $\;\;\;\rightarrow$ $X(t_1), X(t_2), ..., X(t_5)$

($t$ is for "trial number"). Well, this clearly doesn't work. What about this:

1 person: 1, 3, 1, 4, 2 $\;\;\;\rightarrow$ $X(\omega_1), X(\omega_3), ..., X(\omega_2)$
2 person: 2, 2, 1, 6, 3 $\;\;\;\rightarrow$ $X(\omega_2), X(\omega_2), ..., X(\omega_3)$
3 person: 1, 4, 2, 1, 3 $\;\;\;\rightarrow$ $X(\omega_1), X(\omega_4), ..., X(\omega_3)$

This seems to work, but how to write it in a general manner?

person ?: ?, ?, ?, ?, ? $\;\;\;\rightarrow$ $X(?), X(?), X(?), X(?), X(?)$

I think it is this point at which we arrive to an appropriate general notation specified in the option 1.

• You (Leo) did not specify whether you considered that, in the end, your question was answered (I was unable to determine this from your post above) or if you still waited for something. Here the bounty does not help.
– Did
Aug 2, 2011 at 8:53
• @Didier, yes, the answer that I posted is good enough for me. I hope it may be helpful for other people as well.
– Leo
Aug 2, 2011 at 17:47

The first line is a notation for "$n$ given functions $X_i$, all computed on the same input (which was randomly chosen) $\omega$". For example, $\sin(t), \cos(t), \exp(t), \dots$ where $t$ is a randomly chosen integer between 1 and 128. Total information in this set of random numbers is 7 bits no matter how large $n$ is. The sine, cosine, and exponential of $t$ are not independent.

The second line describes: "one function $X$ computed on $n$ different random inputs $\omega_i$". For example, $X(t)$ is the $t^{\rm th}$ user in the math.SE userlist where $t$ is an integer from 1 to 128. If you make $n$ random choices of $t_i$ the amount of information in the list $X(t_1), \dots, X(t_n)$ is $7n$ bits; each $X(t_i)$ is independent of the others.

Only the second corresponds to an i.i.d random sample of $n$ objects. To describe such a sample in the notation used in (1), $\omega$ must depend on $n$, such as $\omega = (\omega_1, \dots, \omega_n)$. The notation as given in the question does not explicitly include any such dependence.

[added in light of the comments discussion]

The original question did not specify precisely what $X$ and $\omega$ mean:

Mathematical description of a random sample: which one is it and why?

1. $X_1(\omega), X_2(\omega), ..., X_n(\omega)$, where $X_1, ..., X_n$ are different but i.i.d. random variables.

2. $X(\omega_1), X(\omega_2), ..., X(\omega_n)$, where $X$ is a (single) random variable.

I think the trouble is that (1) should be written as:

" 0. $X_1, X_2, \dots X_n$ where the $X_i$ are different but i.i.d random variables." (Each $X_i$ is a selection from the distribution of one sample.)

In the original schemes 1-2, writing both "$X$" and "$\omega$" expressions has the implication that they play different roles: that $\omega$ are the random choices made in constructing the samples, and the $X$ or $X_i$ are deterministic functions describing how to convert the random choices into particular samples. For example, if the random samples are single random bits generated using dice, $\omega$ or $\omega_i$ can denote random throws of dice and $X$ a method of converting 1/2/3/4/5/6 outcomes into 0/1 bits. Then $\omega_i$ is itself a sequence of i.i.d random variables, and as a result, so is $X(\omega_i)$, as in formulation (2).

The standard mathematical descriptions are (0), if the details of the sample-generation process are suppressed, and (2), if they are made more explicit. It would be interesting to examine the literature on this but I think (1) is much less common, and where used it may involve strange conventions such as a "prophetic" $\omega$ that includes all future randomness that is used in generating other random variables that are combined with the $X_i$ in subsequent calculations. Be that as it may, there is also a notational or conceptual inconsistency in scheme (1), where the sampling results $X$ are individuated into particular $X_i$ but the underlying sequence of i.i.d random choices is aggregated into one collective $\omega$. The definition of $\omega$ is either left nebulous ("all randomness used to generate the samples") or, if it is made more explicit using the sequence $\omega_i$, then $\omega$ itself is superfluous and the situation would normally be expressed using scheme (2).

The conclusion (or mine at least) is that (1) is defective but (2) or (0) make sense.

• Sorry but the two first sentences of the last paragraph are spectacularly wrong.
– Did
Aug 2, 2011 at 8:56
• The first sentence is a judgement about notational conventions, but the second is an unambiguous mathematical assertion. Could you be kind enough to display the error (e.g. for samples based on discrete finite random choices, as exemplified above)? A random sample of $n$ objects has information content that grows with $n$, do you agree? In scheme (1) the randomness is denoted by Omega whereas the X(i) are deterministic quantities (predetermined functions on the sample space). How can Omega, and thus its entropy, be independent of $n$ in your analysis?
– zyx
Aug 2, 2011 at 17:10
• My first comment was probably abrupt (sorry) but it is a fact that the (universally adopted) way to introduce i.i.d. samples is option 1. Hence the first sentence (far from concerning notational conventions) has (unfortunately) a definite truth value. As regards the second sentence, one cannot open a probability textbook and not read a formulation like Let $(X_n)_{n\ge1}$ denote an i.i.d. sequence of random variables (infinite sequence). If what you write was true (that Omega should depend on the number of random variables considered), this formulation would be nonexistent. Ergo?
– Did
Aug 2, 2011 at 17:31
• In that language, the formulation of (2) is that "the samples $Y_i$ are an i.i.d sequence of random variables, where $Y_i = X(\omega_i)$ (and $i$ goes from 1 to $n$)". There is no way of expressing this in scheme (1) without introducing a dependence of $\omega$ on $n$, presumably $\omega = (\omega_1, \dots, \omega_n)$. Do you agree?
– zyx
Aug 2, 2011 at 17:45
• True, if one tries to express option (2) in the frame of option (1), strange things may happen. But who would want to do that? Not me, and you might have noticed I am not a big fan of option (2), to say the least... Which brings you back to my first comment.
– Did
Aug 2, 2011 at 18:47

The reason why the answer might be (2) rather than (1) is as follows -

To start with we have sample space $$\varOmega$$ with unspecified event space and probability measure. And we have random variable $$X:\varOmega\rightarrow\mathbb{R}$$.

For any $$n$$ we can define a product space and use our random variable to define $$X^{n}:\varOmega^{n}\rightarrow\mathbb{R}^{n}$$ Then a sample $$\overline{\omega}=(\omega_{1},...,\omega_{n})$$ is an element of product sample space $$\varOmega^{n}$$. Here we pinned down all moving parts, there are no unspecified processes or hidden distributions lurking beneath.

However if we assume that n i.i.d. variables are being picked to generate a new sample from the same element $$\omega$$ we don't really pin the definition of the sample because we don't say anything about where those variables came from and how they were picked. It's like saying that we have random variable $$X$$, which we would like to study so postulate n more random variables distributed using same distribution, which we don't know... In general we should try explaining things using a minimal number of existing building blocks and without postulating new building blocks if we can help it.

Another piece of evidence in favour of (2) is that the samples $$X_i$$ are not just distributed identically to $$X$$ but have exactly same values as $$X$$. Sampling means picking values that $$X$$ takes. If $$X_i$$ were only distributed in the same way we could expect them to be able to take values that $$X$$ does not take on null events. But it does not seem to be the case. This suggests that samples are drawn from the sample space rather than returned by n additional i.i.d. random variables working on same $$\omega$$ elements.

There is a puzzling issue with all this, which I am currently struggling with - the samples defined in this way must be equally probable. Does it mean that each $$\omega_i$$ must have same probability? Does it mean that $$\varOmega$$ must be finite and the probability measure on it must be uniform? Or it can be finite and not uniform but we would use weights to equalise chances of all $$\omega_i$$. What about infinite sample spaces? Are they not samplable in a meaningful way?