Suppose we have a sequence of random variables $X^M$ which converges almost surely to a random variable $X^0$ and let $(X_1^M, \ldots, X_M^M)$ be iid samples from $X^M$ for $M \in \mathbb{N}$.

Under which conditions does the law of large numbers hold uniformly in the sense that

$$ |M^{-1} \sum_{m=1}^M X_m^M - \mathbb{E}(X^0)| \xrightarrow{M \to \infty} 0 \; \text{ almost surely?}$$

I am happy to assume that the $X^m$ are uniformly bounded, i.e., that there is a constant $K$ such that

$$ |X^m| \leq K, \; \text{almost surely for all } m \in \mathbb{N}_0.$$

I looked into uniform LLNs but they generally do not seem to fit the setting above.

  • $\begingroup$ Under the condition that $X^m$ have integrable dominant, this follows from Uniform law of large numbers. Please write if you have problems applying it. $\endgroup$
    – zhoraster
    Jun 3 '19 at 20:13
  • $\begingroup$ @zhoraster: I looked at these kind of results. In the case above, the discrete parameter $M$ would probably take the role of $\theta$. However, I have problems with the condition that $M \mapsto X^M(\omega)$ is continuous, since it is only defined at the natural numbers. It would be great if you could provide more details. $\endgroup$
    – dawo
    Jun 4 '19 at 5:04
  • 1
    $\begingroup$ @zhoraster: What could maybe work is to define the function $f(\omega, M^{-1}) = X^M(\omega)$. Then $\lim_{M\to \infty} f(\omega, M^{-1}) = f(\omega, 0) = X^0(\omega)$ as required. To extend the function to a continuous function on [0,1] one could linearly interpolate between the points $M^{-1}$. Is that what you had in mind? $\endgroup$
    – dawo
    Jun 4 '19 at 5:10
  • $\begingroup$ You don't need to have $[0,1]$. Any compact will do. $\endgroup$
    – zhoraster
    Jun 4 '19 at 14:45
  • $\begingroup$ Any luck proving this? $\endgroup$
    – zhoraster
    Jun 8 '19 at 7:52

Assuming uniform integrability of $X^m$, we have that $\mathrm E[X^m]\to \mathrm E[X]$, $m\to\infty$. Therefore, it is enough to show that $$ \left| (\overline{X^m})_m - \mathrm E[X^m]\right|\to 0, m\to \infty,\tag{1} $$ almost surely, where $(\overline{X^m})_m = \frac1m \sum_{i=1}^m X_i^m$.

One possibility is to go through concentration inequalities. For example, if the variables are bounded, as in your question, then by the Hoeffding inequality, for any $\varepsilon>0$, $$ \mathrm P\left(\left| (\overline{X^m})_m - \mathrm E[X^m]\right|>\varepsilon\right)\le e^{-C \varepsilon^2 m} $$ with some $C>0$. Using the Borel-Cantelli lemma, we easily get $(1)$.

Another possibility is, as I commented, to deduce the uniform convergence $$ \sup_m \left| (\overline{X^m})_n - \mathrm E[X^m]\right|\to 0, n\to \infty,\tag{2} $$ from the uniform law of large numbers. However, it seems unlikely that the almost sure convergence can be shown this way; I will only outline the convergence in probability.

Let $F^m$ be the cdf of $X^m$ and $Q^m(t) = \sup\{x\in \mathbb R: F^m(x)<t\}, t\in(0,1)$, be its quasi-inverse (quantile function). Then, as it is well known, $X^m \overset{d}{=} Q^m(U)$, where $U$ is a uniform $[0,1]$ variable. Therefore, $$ (\overline{X^m})_n \overset{d}{=} \frac1n \sum_{k=1}^n Q^m(U_k), $$ where $U_1,U_2,\dots$ are iid uniform $[0,1]$ variables. Also it follows from the weak convergence of $X^m\to X^0$ that $Q^m\to Q^0$ pointwise in the continuity points of $Q^0$, hence, almost everywhere on $(0,1)$.

Now let $\Theta = \{m^{-1}, m\ge 1\}\cup \{0\}$ and set $f(t,m^{-1}) = Q^m(t)$, $m\ge 1$, $f(t,0) = Q^0(t)$. Then, as is explained above, $f(t,\theta)$ is continuous in $\theta$ for almost all $t$ (modulo the distribution of $U$). Therefore, assuming existence of integrable majorant of $f(U,m^{-1})=Q^m(U)$ (which is easily seen to be equivalent to uniform integrability of $X^m$), we get that $$ \sup_{\theta\in \Theta}\left| \frac1n \sum_{k=1}^n f(U_k,\theta) - \mathrm{E}[f(U,\theta)]\right| \to 0, n\to \infty, $$ almost surely, whence we get the convergence $(2)$ in probability (remember that we replaced $(\overline{X^m})_n$ by its distributional copy).

The convergence in probability might sound bad, but there are at least two advantages:

  1. Only uniform integrability is required.

  2. The approach works for any $(n_m,m\ge 1)$ such that $n_m\to\infty$, $m\to\infty$, i.e. we have $$ \left| (\overline{X^m})_{n_m} - \mathrm E[X^m]\right|\to 0, m\to \infty, $$ in probability. The first approach fails (to establish the almost sure convergence) for "small" $n_m$.

  • $\begingroup$ Thank you for this extensive answer! If I understand your point correctly, both approaches 'only' yield convergence in probability. You might be right that in the general setting outlined above, almost sure convergence will be hard to establish. I also largely agree with your assessment of convergence in probability. However, my problem has a little more structure and the original post was written to abstract from my concrete situation. In the process, I might have lost what actually might help to establish almost sure convergence (see post below). $\endgroup$
    – dawo
    Jun 19 '19 at 5:28
  • $\begingroup$ @dawo, no, the first approach gives an a.s. convergence. $\endgroup$
    – zhoraster
    Jun 20 '19 at 3:28
  • $\begingroup$ Thank you - I must have missed the Borel-Cantelli Lemma on the first reading. $\endgroup$
    – dawo
    Jun 26 '19 at 13:39

In the case I am considering the random variable $X^M$ can actually be written as function of $X^0$ and $M$, i.e., $X^M = f(X^0, M)$ for all $M \in \mathbb{N}$.

A priori, this function is only defined on the integers. However, I could extend it to a function $f: \mathbb{R} \times [0,1] \to \mathbb{R}$ by defining $$ f(x, \theta) = \begin{cases} f(x,M), & \text{if } \theta = M^{-1}, \text{ for } M \in \mathbb{N} \\ \frac{\lambda}{M^{-1}-(M+1)^{-1}} f(x,M) + \frac{1-\lambda}{M^{-1}-(M+1)^{-1}} f(x,M+1), & \text{for } \theta = \frac{\lambda M^{-1}}{M^{-1}-(M+1)^{-1}} + \frac{(1-\lambda)(M+1)^{-1}}{M^{-1}-(M+1)^{-1}} \end{cases}.$$

By my assumption on the almost sure convergence $X^M \to X^0$, it follows that $$ f(x, \theta) \xrightarrow{\theta \to 0} f(x, 0)$$ and the continuity of $f$ at all other points $\theta \in (0,1]$ follows from the way $f$ is defined.

Hence, in this case the conditions for the uniform LLN are fulfilled and it would follow that $$\sup_{\theta \in [0,1]} \left| n^{-1} \sum_{i=1}^n f(X_i^0, \theta) - \mathbb{E} (f(X^0, \theta)) \right| \xrightarrow{n \to \infty} 0 \quad \text{almost surely.}$$ This would imply $$\sup_{M \in \mathbb{N}} \left| n^{-1} \sum_{i=1}^n f(X_i^0, M^{-1}) - \mathbb{E} (f(X^0, M^{-1})) \right| = \sup_{M \in \mathbb{N}} \left| n^{-1} \sum_{i=1}^n X_i^M - \mathbb{E} (X_i^M) \right| \xrightarrow{n \to \infty} 0 \quad \text{almost surely.}$$

Hence, for every $\epsilon > 0$, there is a $N(\epsilon)$ such that $$\left| n^{-1} \sum_{i=1}^n X_i^M - \mathbb{E} (X_i^M) \right| < \epsilon, \quad \forall n \geq N(\epsilon),$$ which means that for $M \geq N(\epsilon)$, we have $$\left| M^{-1} \sum_{i=1}^M X_i^M - \mathbb{E} (X_i^M) \right| < \epsilon \quad \text{almost surely.}$$

As mentioned in the post by zhoraster above, $$ |\mathbb{E}(X^M) - \mathbb{E}(X^0)| \xrightarrow{M \to \infty} 0 \quad \text{almost surely}$$ using the boundedness assumption on the $X^M$ (or indeed the weaker condition of uniform integrability).

Putting these two things together establishes (via an application of the triangle inequality) that $$ |M^{-1} \sum_{m=1}^M X_m^M - \mathbb{E}(X^0)| \xrightarrow{M \to \infty} 0 \; \text{ almost surely.}$$

  • $\begingroup$ As I said, considering piecewise linear functions is overcomplicating things. If indeed $F(X^0,M)\to X_0$, $M\to\infty$, a.s., then you have a strong (i.e. almost sure) LLN provided that $\sup_{M\ge 1} F(X^0,M)$ is integrable (uniform integrability of $F(X^0,M)$ might be enough). Just take $\Theta$ as in my answer. $\endgroup$
    – zhoraster
    Jun 20 '19 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.