Is all of probability theory “derivable” from Bernoulli trials?

Question

I once heard an offhand remark that all of probability theory is "derivable" (whatever that means) from Bernoulli trials. Is there any truth to this statement?

I might imagine that, say, all probability distributions are sufficiently approximable by an infinite sequence of coin flips with varying weights. Or that all theorems of probability theory can be proved just within the context of Bernoulli trials (this doesn't seem all that enlightening).

Can anyone point to a well-known theorem (or common knowledge) that elaborates on this?

Answer 1

I think the idea is, stated loosely, as follows:

Suppose you are given some random variable $X$. Then using only the "randomness" of a countable sequence of fair coin flips $$f_i: \Omega_i \rightarrow \mathbb{R} \quad \mathbb{P}(f_i = 1) = \mathbb{P}(f_i = 0) = \frac{1}{2}$$you can produce a random variable $X'$ with the same distribution function as $X$.

Indeed, suppose first that $X$ was uniform on $[0,1]$, then you can use $$G: \otimes_i \Omega_i \rightarrow \mathbb{R} \quad X' = \sum \frac{f_i}{2^i}$$Loosely, all we're saying is that being uniformly distributed on $(0,1)$ is the same as having all your digits be fair coin flips in base 2.

From here, it's just trickery. Namely, if $X$ has distribution function $F: \mathbb{R} \rightarrow [0,1]$, consider the function $$g: [0,1] \rightarrow \mathbb{R} \quad t \mapsto \sup \{x \in \mathbb{R}: \; F(x) \leqslant t\}$$If $F$ is reasonable (say continuous), then we have that $g \circ G$ has the same distribution function as $X$ by construction.

(there's something annoying about $g$ being $\infty$ valued at $1$, and being the sup of the empty set at 0, but this is no big deal)

Answer 2

Consider a couple special cases of multiple Bernoulli trials: one where $p$ remains fixed, and one where $np$ remains fixed as $n \to \infty$ ($p$ represents the weighted probability of the trial, and $n$ represents the number of trials).

We have a few theorems that help us convert this setup into some well-known distributions. In the former case, we have the de Moivre-Laplace Theorem which leads to a normal distribution; in the latter case, we have the Poisson Limit Theorem which leads to a Poisson distribution.

Consider the first scenario, since your question appears to want to talk about approximating continuous distributions with Bernoulli trials (if you want to talk about discrete distributions, the discussion is not much different). In such a case, the de Moivre-Laplace Theorem gives us a fairly easily understood way to obtain a specific continuous distribution from discrete trials. But this is pretty far from approximating every distribution in such a manner.

Norbert Wiener used the Cameron-Martin Theorem to show that we can approximate arbitrary distributions with a normal distribution under the condition that the target distribution has finite second moment (equivalently, that it is mean-squared convergent). Later work extended this to other well-known distributions: uniform, beta, gamma, exponential, etc. Likewise, discrete distributions can be approximated in a similar way.

So, in a sense, we can definitely approximate -- or derive in a limiting sense -- mean-squared convergent distributions from Bernoulli trials, given some conditions on $p$.

I am not sure what happens when we remove this restriction. I would imagine it would be difficult, if not impossible, to approximate some distributions that do not have finite second moment (e.g. the Cauchy distribution) in this way.

Answer 3

I don't think there exist a well know theorem about this. Although you may know, that every random variable is a measurable function, and the measurable function can be discrete or continue. Continue mesurable function are limit's approximation of discrete (simple) measurable function.

Maybe this links (last page) will be helpful: probability and statistics cookbook As you can see, the only entering arrow for Bernoulli's trial is from the Binomial (that's derived from repeating bernoulli's trial).

I should have been more precise, but i'm still preparing this exams.

p.s. i love questions linke this, those answer is (imho) very meaningful in the deep understanding of a topic