# Is there a distribution over all distributions in [0,1]?

Let's call $$D$$ to the set of all distributions over the interval $$[0,1]$$, including the uniform, the triangular, the Dirac-delta at $$0$$, and in general any measure $$\mu$$ such that $$\mu([0,1])=1$$.

We can define several types of distributions over $$D$$. For example, provided a sequence of distributions $$\mu_1, \mu_2, ...\in D$$, we could just draw a random index $$k$$ using a fixed distribution over $$\{1,2,...\}$$ and output $$\mu_k \in D$$. Similarly as a second example, provided a family of distributions $$\mu_\theta$$ parametrized by a vector $$\theta\in\mathbb R^d$$, we can draw $$\theta$$ from a fixed distribution over $$\mathbb R^d$$ and output $$\mu_\theta \in D$$. But in both examples, we are restricted to a fixed family of distributions ($$\mu_i$$ or $$\mu_\theta$$). I wonder if there is an "unrestricted" distribution over $$D$$ that "captures all distributions in $$D$$".

Can you provide an example of a distribution $$\phi$$ over $$D$$ whose support is $$D$$?

To define support formally, use the earth's mover distance, or any Wasserstein metric, to define the distance between any two distributions $$\mu,\mu'\in D$$. Since these distances are metrics for $$D$$, they define a topology over $$D$$ and enable the notion of support of any distribution $$\phi$$ over $$D$$.

• You can always define such a measure/distribution on any set, but whether it is useful is another question. Dec 6, 2021 at 18:20
• A CDF $F(x)$ is completely specified by its values $F(r)$ for all rational numbers $r$. So the set of all CDF functions has cardinality $|\mathbb{R}^{\mathbb{Q}}|$, and $|\mathbb{R}^{\mathbb{Q}}|=|\mathbb{R}|=|(0,1)|$. So there is a bijection between the set of CDFs and the unit interval $(0,1)$, call it $b:(0,1)\rightarrow \{\mbox{all CDFS}\}$. So you can generate a uniformly distributed random variable $U \sim Unif(0,1)$ and then $b(U)$ is a randomly chosen CDF, each possible CDF being equally likely. Dec 7, 2021 at 15:25
• @Michael The phrase "each possible CDF being equally likely" is a little misleading, since the distribution you get depends critically on the choice of bijection. The "uniformness" of the uniform distribution doesn't carry through.
– Karl
Dec 7, 2021 at 16:43
• "...each possible CDF being [a possible result] (and equally likely)." Of course the probability is (equally) zero for each particular CDF. Equally likely does not mean uniformly distributed (any continuous random variable has equally likely values), though, if you want to critique my random distribution as not being "uniform" you might want to define what you mean by "uniform" over the space of all CDFs. Dec 7, 2021 at 16:55
• Thank you both, the conclusion is that there is such a distribution. Can any of you provide constructive example that explicitly shows the bijection? Dec 7, 2021 at 17:31

Here is a structured form of my above comment: We sequentially build a CDF $$F(x)$$ by choosing its values on rationals.

We seek CDFs $$F(x)$$ such that $$F(x)=0$$ for $$x<0$$ and $$F(x)=1$$ for $$x\geq 1$$. We shall specify $$F(q)$$ for all rational numbers $$q \in [0,1]$$. Let $$\{q_0=1, q_1=0, q_2, q_3, q_4, ...\}$$ be a listing of all rationals in $$[0,1]$$ (starting the list with $$1$$ and then $$0$$).

1. Choose $$X_0=1$$ and define $$F(1)=X_0=1$$.

2. Choose $$X_1\sim U[0,1]$$ and define $$F(0)=X_1$$.

3. To enforce monotonicity, for each $$i \in \{2, 3, 4, ...\}$$ define the finite list $$A_i = \{q_0=1, q_1=0, q_2, ..., q_{i-1}\}$$ Since $$0\in A_i$$, at least one rational in $$A_i$$ is smaller than $$q_i$$. Define $$l_i$$ as the index of the largest rational in $$A_i$$ that is smaller than $$q_i$$. Similarly, since $$1 \in A_i$$, we can define $$r_i$$ as the index of the smallest rational in $$A_i$$ that is larger than $$q_i$$. Thus: $$q_{l_i} < q_i < q_{r_i} \quad \forall i \in \{2, 3, 4, ...\}$$ Define the lower and upper bounds: $$L_i = X_{l_i}, U_i = X_{r_i}$$ If $$L_i=U_i$$ define $$X_i=L_i$$ and $$F(q_i)=X_i$$. If $$L_i then independently choose $$X_i \sim U[L_i, U_i]$$ and define $$F(q_i)=X_i$$.

Here we have built random values for $$F(q_i)$$ for all $$i \in \{0, 1, 2, ...\}$$. By construction the values satisfy monotonicity and satisfy $$0\leq F(0)\leq F(1)=1$$. For each $$i$$, with probability 1, right-continuity holds at $$q_i$$, that is $$F(q_j)\rightarrow F(q_i)$$ over any sequence of rationals $$\{q_j\}_{j \in J}$$ that converge to $$q_i$$ from the right. So this property holds for all $$q_i$$ with prob 1. Throw away any sample path for which right-continuity does not hold and try again (this happens with prob 0). Now define $$F(x)$$ for irrational $$x \in [0,1]$$ by taking right-limits over rationals (the right-limits exist by monotonicity). The result is a valid CDF.

You can use the distance between two CDFs $$F(x)$$ and $$G(x)$$ by $$d(F,G)= \sum_{i=0}^{\infty}2^{-(i+1)}|F(q_i)-G(q_i)|$$ Fix $$\epsilon>0$$. Fix any CDF $$G(x)$$ that satisfies $$G(x)=0$$ if $$x<0$$ and $$G(x)=1$$ if $$x\geq 1$$. If we randomly choose $$F(x)$$ according to the above procedure then $$P[d(F,G)<\epsilon]>0$$ To see this, fix a positive integer $$n\geq 3$$ such that $$\sum_{i=n+1}^{\infty} 2^{-(i+1)} < \epsilon/2$$. Then $$\{d(F,G)<\epsilon\} \supseteq \bigcap_{i=1}^n \{|X_i-G(q_i)|<\epsilon/2\}$$

• Why is it problematic when right continuity does not hold at a rational? I think we should leave that case untouched because it is the only way to produce discrete distributions over the rationals, e.g. a Dirac delta at one half. Dec 8, 2021 at 15:31
• Left-discontinuity is fine but right-continuity must hold for all CDF functions. A function $F:\mathbb{R}\rightarrow\mathbb{R}$ is a valid CDF if and only if it is nondecreasing, right-continuous, and satisfies $\lim_{x\rightarrow\infty} F(x)=1$ and $\lim_{x\rightarrow-\infty} F(x) = 0$. Dec 8, 2021 at 18:11