# The measure-theoretical definition of a bootstrap sample

I’m currently learning the bootstrap method, and I have two questions to ask about the definition of a bootstrap sample.

Let $(\Omega,\mathscr{S},\mathsf{P})$ be a probability space. Let $X_{1},\ldots,X_{n}$ be i.i.d. random variables on $(\Omega,\mathscr{S},\mathsf{P})$, with their common c.d.f. denoted by $F$. Let $\hat{F}$ denote the empirical c.d.f. of $X_{1},\ldots,X_{n}$, i.e., $$\forall x \in \mathbf{R}: \qquad \hat{F}(x) = \frac{1}{n} \sum_{i = 1}^{n} \chi_{(- \infty,x]} \circ X_{i}.$$ Clearly, $\hat{F}(x)$ is a random variable on $(\Omega,\mathscr{S},\mathsf{P})$ for each $x \in \mathbf{R}$, and for each $\omega \in \Omega$, the function $$\left\{ \begin{matrix} \mathbf{R} & \to & [0,1] \\ x & \mapsto & \left[ \hat{F}(x) \right] \! (\omega)\end{matrix} \right\}$$ is the c.d.f. of some discrete random variable.

Question 1: What does it mean to say that $(X_{1}^{*},\ldots,X_{n}^{*})$ is a bootstrap sample drawn from $\hat{F}$? As mentioned, $\hat{F}(x)$ is not a number but a random variable for each $x \in \mathbf{R}$. I require an answer to this question strictly in terms of measure theory.

Question 2: What probability space are $X_{1}^{*},\ldots,X_{n}^{*}$ defined on? Is it still $(\Omega,\mathscr{S},\mathsf{P})$?

Thanks!

I’ve managed to answer my questions. In what follows, we fix $n \in \mathbf{N}$ and denote $\mathbf{N}_{\leq n}$ by $[n]$.

Let $\mathcal{R}$ denote the set of random variables defined on the probability space $([n]^{n},\mathcal{P}([n]^{n}),\mathsf{c})$, where $\mathsf{c}$ denotes the probability measure on $([n]^{n},\mathcal{P}([n]^{n}))$ having a mass of $\dfrac{1}{n^{n}}$ at every element of $[n]^{n}$. Then $X_{1}^{\ast},\ldots,X_{n}^{\ast}$ are $\mathcal{R}$-valued functions on $\Omega$ such that for any $i \in [n]$ and $\omega \in \Omega$, the following conditions hold:

• ${X_{i}^{\ast}}(\omega): [n]^{n} \to \{ {X_{k}}(\omega) \}_{k \in [n]}$.
• $[{X_{i}^{\ast}}(\omega)](\mathbf{a}) = {X_{\mathbf{a}(i)}}(\omega)$ for each $\mathbf{a} \in [n]^{n}$.

One can easily verify that for each $\omega \in \Omega$, the c.d.f. of ${X_{i}^{\ast}}(\omega)$ for any $i \in [n]$ is precisely $\left[ \hat{F}(\cdot) \right] \! (\omega)$.

If $\displaystyle \bar{X} \stackrel{\text{df}}{=} \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ and $\displaystyle \bar{X}^{\ast} \stackrel{\text{df}}{=} \frac{1}{n} \sum_{i = 1}^{n} X_{i}^{\ast}$, then $\bar{X}^{\ast} - \bar{X}$ is to be interpreted as an $\mathcal{R}$-valued function on $\Omega$, i.e., $$\forall \omega \in \Omega: \qquad \left( \bar{X}^{\ast} - \bar{X} \right) \! (\omega) = \frac{1}{n} \sum_{i = 1}^{n} {X_{i}^{\ast}}(\omega) - \underbrace{\frac{1}{n} \sum_{i = 1}^{n} {X_{i}}(\omega)}_{(\star)},$$ where the term $(\star)$ is viewed as a constant random variable on $([n]^{n},\mathcal{P}([n]^{n}),\mathsf{c})$.

Here is a complementary (but equivalent) construction of the bootstrap sample:

Let $(\Omega_1,\Sigma_1,P_1)$ be a probability space and consider the $n$-fold product space $(\Omega^S,\Sigma^S,P^S)$ with coordinate maps $X_1,\ldots,X_n$. In other words, $X_1,\ldots,X_n$ is the canonical choice of of i.i.d. random variables with law $P_1$, and $(\Omega^S,\Sigma^S,P^S)$ is the space of our original sample.

Next, consider the probability space $(\Omega_2,\Sigma_2,P_2)$, where $\Omega_2=\{1,\ldots,n\}$ is the $n$-point set, $\Sigma_2$ is the power set $\sigma$-algebra and $P_2$ is the uniform measure. Let $(\Omega^B,\Sigma^B,P^B)$ denote the $n$-fold product of this space with coordinate maps $\tau_1,\ldots,\tau_n$. Thus, $\tau_1,\ldots,\tau_n$ are i.i.d. uniformly distributed on $\{1,\ldots,n\}$.

Finally, let $(\Omega,\Sigma,P)$ denote the product of $(\Omega_1,\Sigma_1,P_1)$ and $(\Omega_2,\Sigma_2,P_2)$. The variables $X_1,\ldots,X_n$ can be viewed as variables in $(\Omega,\Sigma,P)$ by putting $X_j(\omega):=X_j(\omega^S)$ for $\omega=(\omega^S,\omega^B)$, and similarly for $\tau_1,\ldots,\tau_n$.

Definition: The bootstrap sample $X^*_1,\ldots,X^*_n$ of $X_1,\ldots,X_n$ is defined by $X^*_j:=X_{\tau_j}$.

That is, $X^*_j(\omega)=X_{\tau_j(\omega)}(\omega)=X_{\tau_j(\omega^B)}(\omega^S)$, or equivalently $X^*_j=\sum_{k=1}^n X_k1_{(\tau_j=k)}$.

Since $P(\tau_j=k)=1/n$ and $X_1,\ldots,X_n,\tau_1,\ldots,\tau_n$ are independent, we see that almost surely $$E(1_A(X^*_j)\mid X_1,\ldots,X_n)=\frac{1}{n}\sum_{k=1}^n E(1_A(X_k)\mid X_1,\ldots,X_n) = \frac{1}{n}\sum_{k=1}^n 1_A(X_k) = \frac{1}{n}\sum_{k=1}^n \delta_{X_k}(A).$$ This ensures, as expected, that the empirical measure $P_n(A,\omega):=\frac{1}{n}\sum_{j=1}^n \delta_{X_j(\omega)}(A)$ is a conditional distribution for $X_j^*$ given $X_1,\ldots,X_n$. Similarly, we have almost surely \begin{align} &E(1_{A_1}(X^*_1)\cdots 1_{A_n}(X^*_n)\mid X_1,\ldots,X_n) \\ &=\frac{1}{n^n}\sum_{k_1=1}^n\cdots \sum_{k_n=1}^n 1_{A_1}(X_{k_1})\cdots 1_{A_n}(X_{k_n})\\ &=E(1_{A_1}(X^*_1)\mid X_1,\ldots,X_n)\cdots E(1_{A_n}(X^*_n)\mid X_1,\ldots,X_n), \end{align} which ensures that $X_1^*,\ldots,X_n^*$ are conditionally independent given $X_1,\ldots,X_n$.

Dudley, R.M., Real analysis and probability., Cambridge Studies in Advanced Mathematics. 74. Cambridge: Cambridge University Press. x, 555 p. (2002). ZBL1023.60001.

Dudley, R. M., Uniform central limit theorems, Cambridge Studies in Advanced Mathematics 142. Cambridge: Cambridge University Press (ISBN 978-0-521-73841-5/pbk; 978-0-521-49884-5/hbk; 978-1-139-01483-0/ebook). 482 p. (2014). ZBL1317.60030.

Question 1: By bootstrap sample we mean a sample, which we get by choosing $n$ times with replacement from $X_1,..., X_n$.

$\forall k\in \{1,2,..., n\}$

$$X^{*}_{k} : \left(\begin{array}{cccc} X_1 & X_2 & ... & X_n\\ \frac{1}{n} & \frac{1}{n} & ... & \frac{1}{n}\end{array}\right)$$

Note that it can happen that any of this $X_i$ and $X_j$ in their outcomes could have same numbers (if we have discrete r.v. in the start).

Question 2: We should see that $X:\Omega\to \mathbb{R}$, $X^{*}:\Omega_1\to \{x_1,..., x_n\}$, because:

for each $\omega \in \Omega$ and $\forall i\in \{1,..., n\}$ we get $X_i(w)=x_i$, then: $$X^{*}_{i} : \left(\begin{array}{cccc} x_1 & x_2 & ... & x_n\\ \frac{1}{n} & \frac{1}{n} & ... & \frac{1}{n}\end{array}\right)$$ Hence the $\Omega$ is here for getting our $X_i^*$ values, or precisely it "helps" $X_i^*$ to become exact random variables. Then we need some new set $\Omega_1$ for getting outcomes from $X_i^*.$ Hence sigma-algebra is also new one, just as the probability.

Some notes: again, what is bootstrap really doing. First we get sample $X_1,..., X_n$ from our original set, then we are sampling this sample, which we call resampling.

Hope it will help.

• Thanks for your response. However, it leaves me more confused than before. Some texts, such as Measure Theory and Probability by Athreya and Lahiri, talk about objects like $\bar{X}^{\ast} - \bar{X}$, where $\displaystyle \bar{X} \stackrel{\text{df}}{=} \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ and $\displaystyle \bar{X}^{\ast} \stackrel{\text{df}}{=} \frac{1}{n} \sum_{i = 1}^{n} X_{i}^{\ast}$. If this is to make sense, then the $X_{i}^{\ast}$’s must be defined on the original probability space $(\Omega,\mathscr{S},\mathsf{P})$. Commented Apr 19, 2017 at 19:55
• I’ve gotten it! Let $\mathcal{R}$ denote the set of random variables on $([n]^{n},\mathcal{P}([n]^{n}),\mathsf{c})$, where $\mathsf{c}$ denotes the probability measure on $[n]^{n}$ with mass $\dfrac{1}{n^{n}}$ on each element of $[n]^{n}$. Fix any $i \in [n]$. Then $X_{i}^{\ast}$ is an $\mathcal{R}$-valued function on $\Omega$ such that for any $\omega \in \Omega$, we have (i) ${X_{i}^{\ast}}(\omega): [n]^{n} \to \{ {X_{k}}(\omega) \}_{k \in [n]}$ and (ii) $[{X_{i}^{\ast}}(\omega)](\mathbf{a}) = {X_{\mathbf{a}(i)}}(\omega)$ for each $\mathbf{a} \in [n]^{n}$. Commented Apr 20, 2017 at 8:18