Let $X$ and $\varepsilon$ be independent random vectors, $\mathcal{X} = \text{supp}(X)$, and $Y = f(X) + \varepsilon$ for some function $f$. For any $x \in \mathcal{X}$, let $y^i = y^i(\omega)$, $i \in \{1,\cdots,n\}$, be independent samples of $Y \mid X = x$ defined on a measurable space $(\Omega,\mathcal{F})$ and equipped with a probability measure $\mathbb{P}_x$. Suppose for each $x \in \mathcal{X}$, there exists a $\mathcal{F}$-measurable set $S(x)$ such that $\mathbb{P}_x\{S(x)\} = 0$ and for any $\omega\in \Omega \backslash S(x)$, $$\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^{n} g(y^i(\omega)) = \mathbb{E}[g(Y) \mid X = x]$$ for some function $g$.

Question: Can we find a measurable set $T$ such that $\mathbb{P}_x\{T\} = 0$, $\forall x \in \mathcal{X}$, and for any $\omega\in \Omega \backslash T$, $$\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^{n} g(y^i(\omega)) = \mathbb{E}[g(Y) \mid X = x]? \tag{1}$$ If this is not true in general, are there mild assumptions on the functions $f$ and $g$ and/or the distribution of $\varepsilon$ under which it holds?

Note: $\mathcal{X} \subset \mathbb{R}^m$ is the support of $X$, $f: \mathcal{X} \to \mathbb{R}^d$, and $g: \mathcal{Y} \to \mathbb{R}$, where $\mathcal{Y}$ is the support of $Y$.

Context: This question is based on Assumption (A6) in page 11 of this paper. I am not well-versed in measure theory, so forgive me for incorrect use of notation.

Thoughts: My rough interpretation is that $S(x)$ denotes the sample paths of $Y \mid X = x$ of probability zero over which the LLN-type equality does not hold. Generally, this set can depend on $x \in \mathcal{X}$, and the question is whether there exists a set (independent of $x$) $T \supset S(x)$, for a.e. $x \in \mathcal{X}$ also of zero probability for which the equality holds.

Clearly, this holds when $f \equiv 0$ (i.e., $Y$ is independent of $X$) since the set $S$ does not depend on $x$ in this case. When $f$ is not trivially zero, it seems like there cannot be (uncountably) many values that the set $S(x)$ can take, because the conditional distributions $Y \mid X = x_1$ and $Y \mid X = x_2$ only differ by a translation when $x_1 \neq x_2$.

Plausible argument for the case when $f$ is continuous and $g$ is Lipschitz continuous: Let $\bar{\mathcal{X}} = \mathcal{X} \cap \mathbb{Q}^m$ be the intersection of the support of $X$ with $m$-dimensional rational vectors. Then $T = \cup_{x \in \bar{\mathcal{X}}} S(x)$ satisfies (1). I think this is true because if $\omega \in S(x)$ for some $x \in \mathcal{X} \backslash \bar{\mathcal{X}}$, then we can pick $\bar{x} \in \bar{\mathcal{X}}$ that is arbitrarily close to $x$ (since $\mathbb{Q}^m$ is dense in $\mathbb{R}^m$) such that $\omega \in S(\bar{x})$.

  • $\begingroup$ I don't mean to be annoying, but can you get rid of a lot of your question to ask a standalone question? You may certainly provide context, but after you have asked your question. It would be nice if you can ask a general, self-contained, question at the beginning. Like, do you need, $f$, $\epsilon$, etc? I have a headache now, so maybe I'm just too sensitive now to much information. $\endgroup$ – mathworker21 Apr 2 '20 at 23:22
  • $\begingroup$ @mathworker21 No worries! I tried my best to be concise. I'm afraid to change it too much from what's in the paper because I fear I might be imprecise with the notation/formulation. $\endgroup$ – madnessweasley Apr 3 '20 at 3:23
  • $\begingroup$ Thanks! Why do you have to suppose the existence of $S(x)$? Doesn't its existence follow from LLN? $\endgroup$ – mathworker21 Apr 3 '20 at 5:39
  • $\begingroup$ @mathworker21 Yes, I'm essentially assuming that the strong LLN holds for the function $g$ in question $\endgroup$ – madnessweasley Apr 3 '20 at 5:41
  • $\begingroup$ @mathworker21 It isn't obvious to me. The sets $S(x_1)$ and $S(x_2)$ can be different for $x_1 \neq x_2$, but it isn't clear to me that we can't find a single set $T$ of measure zero for which the equality holds for all $x$. For example, the paper says that if $\mathcal{X}$ is a finite set, then the result holds. I presume this is true by considering $T := \cup_{x \in \mathcal{X}} S(x)$. If so, then it seems like this trick would work even if $\mathcal{X}$ is countable. $\endgroup$ – madnessweasley Apr 3 '20 at 5:49

The answer is "yes", assuming $g$ is continuous.

There are two extreme kinds of $\epsilon$. One kind is discrete $\epsilon$ (only atoms), and another is smooth $\epsilon$ (no atoms). I will deal with each, and let you handle mixtures.

Let's first do discrete. The limit condition is $\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n g(f(x)+\epsilon_i) = \sum_{\epsilon'} p(\epsilon')g(f(x)+\epsilon')$, where $(\epsilon_i)_i$ is the sampled $\epsilon$'s and $p(\epsilon')$ is the probability of $\epsilon$ being a specific $\epsilon'$. We may let $T$ be the set of all $\omega$, equivalently the set of all $(\epsilon_i)_i$, such that there is some $\epsilon'$ with $\lim_{n \to \infty} \frac{1}{n}\#\{1 \le i \le n : \epsilon_i = \epsilon'\} \not = p(\epsilon')$ for each $\epsilon'$. It's easy to see $P_x[T] = 0$ for each $x$ (there's no dependence on $x$; $\epsilon$ is an independent thing, so by LLN, we have $\lim_{n \to \infty} \frac{1}{n}\#\{1 \le i \le n : \epsilon_i = \epsilon'\} = p(\epsilon')$ for each $\epsilon'$ with probability $1$).

Now let's do continuous. Let's suppose $\epsilon$ is supported in $0,1$, and $\mu$ is the distribution of $\epsilon$. We want $\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n g(f(x)+\epsilon_i) = \int_0^1 g(f(x)+\epsilon')d\mu(\epsilon')$. We let $T$ be the set of all $(\epsilon_i)_i$ for which there is some subinterval $(a,b) \subseteq (0,1)$ with $\lim_{n \to \infty} \frac{1}{n}\#\{1 \le i \le n : \epsilon_i \in (a,b)\} \not \to \mu((a,b))$. Once again, $P_x[T]$ is independent of $x$, and it is $0$ by LLN. To see that $\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n g(f(x)+\epsilon_i) = \int_0^1 g(f(x)+\epsilon')d\mu(\epsilon')$ for each $(\epsilon_i)_i$ not in $T$, we use continuity of $g$ (this is an easy analysis argument; let me know if you want me to sketch it out).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.