# Strong law of large numbers for function of random vector: can we apply it for a component only?

Consider

• i.i.d. random variables $\{X_1,..., X_n\}$ with well defined first moment

• i.i.d. random variables $\{Y_1,..., Y_n\}$ with well defined first moment

By the strong law of large numbers: $$\frac{1}{n}\sum_{i=1}^n Y_i \rightarrow_{a.s.} E(Y_i) \text{ }\text{ as n\rightarrow \infty}$$

Consider these three objects for any function $g: \mathbb{R}^2\rightarrow \mathbb{R}$ (take $Y_i$ discrete with support $\mathcal{Y}$ for simplicity)

1) for a given realisation $x$ of $X_k$, $E(g(X_k, Y_i)| X_k=x)\equiv \sum_{y\in \mathcal{Y}} g(x, y)P(Y_i=y|X_k=x)$ which is a scalar

2) $E(g(X_k, Y_i)| X_k)\equiv \sum_{y\in \mathcal{Y}} g(X_k, y)P(Y_i=y|X_k)$ which is a random variable because $g(X_k,y)$ and $P(Y_i=y|X_k)$ are both functions of the random variable $X_k$

3) $F(X_k)\equiv\sum_{y\in \mathcal{Y}}g(X_k, y)\mathbb{P}(Y_i=y)$ which is a random variable because $g(X_k, y)$ is a function of $X_k$.

When $X_k\perp Y_i$, then $(2)=(3)$.

Question: 1) Is it true that $\forall k=1,...,n$ $$\frac{1}{n}\sum_{i=1}^n g(X_k, Y_i) \rightarrow_{a.s.} F(X_k) \text{ }\text{ as n\rightarrow \infty}$$ If yes, under which conditions?

2) Is it true that $\forall k=1,...,n$ $$\frac{1}{n}\sum_{i=1}^n g(X_k, Y_i) \rightarrow_{a.s.} E(g(X_k, Y_i)|X_k) \text{ }\text{ as n\rightarrow \infty}$$ If yes, under which conditions?

EDIT: This question here is close to mine and includes also an answer. However it is for $g(X_k, Y_i)=Y_i\times X_k$

• I still don't think I fully understand what you mean by "expectation with respect to $Y_i$ only." Can you give a precise definition? Can it be expressed in terms of some conditional expectation? Apr 26, 2018 at 13:43
• @NateEldredge Thank you, see my edited question
– Star
Apr 26, 2018 at 14:00
• @CGT : The ambiguity mentioned by Nate persists for your use of "$E_{Y_i}[g(X_k,Y_i)]$." As in my comment to the Adrian answer, it would be more clear if you write expectations without subscripts, as in $E[Z]$ or $E[Z|W]$. Apr 26, 2018 at 14:30
• @Michael Thank you. I have edited my notation: is it better now?
– Star
Apr 26, 2018 at 14:36
• @CGT : Yes the notation makes sense now. For question 2 the answer is "yes" if $\{Y_i\}$ is independent of $X_k$ (or more generally, if $\{Y_i\}$ are conditionally i.i.d. given $X_k$). However it is "no" if $g(x,y)=xy$ and $X_i=Y_i$ for all $i$ (which may be what Adrian wants in his answer, I cannot tell). In that case $E[X_kY_i|X_k] = X_kE[Y_i]$ if $i \neq k$, but $E[X_kY_k|X_k]=X_k^2$, so the expression $E[X_kY_i|X_k]$ can change with $i$. Apr 26, 2018 at 15:09

## 2 Answers

Your first question is deeper. Here I show the answer is "yes" under some general conditions, though I expect the answer is "yes" even for more general conditions than I give.

Your question boils down to this: Let $\{Y_i\}_{i=1}^{\infty}$ be i.i.d. and let $X$ be a random variable that possibly depends on $\{Y_i\}_{i=1}^{\infty}$. Define $f:\mathbb{R}\rightarrow\mathbb{R}$ by: $$f(x) = E[g(x,Y_1)]$$ and assume $f(x)$ is finite for all $x \in \mathbb{R}$. We want to know if the following is true: $$\lim_{n\rightarrow\infty} \frac{1}{n} \sum_{i=1}^ng(X, Y_i) = f(X) \quad \mbox{with prob 1}$$

### Why the question is interesting

The random variable $X$ might have information about one (or even all) of the $Y_i$ variables. For example, suppose $\{Y_i\}_{i=1}^{\infty}$ are i.i.d. binary equally likely, and $X = \sum_{i=1}^{\infty} Y_i 2^{-i}$. Then $X$ is uniform over $[0,1]$ and from $X$ we can take a binary decimal expansion to reconstruct each $Y_i$ (assuming the probability 0 event of $\{Y_i\}$ having an infinite tail of 1's does not occur).

### Proof of "yes" when $X$ takes values in a countable set

Suppose $X$ takes values in a finite or countably infinite set $\mathcal{X}$. For each $x \in \mathcal{X}$ define the event $$A_x = \left\{ \lim_{n\rightarrow\infty} \frac{1}{n}\sum_{i=1}^n g(x,Y_i) = f(x) \right\}$$ with $f(x)$ defined by $f(x)=E[g(x,Y_1)]$ for each $x \in \mathcal{X}$. Note that $P[A_x]=1$ for all $x \in \mathcal{X}$ (by the Law of Large Numbers). Define $$A = \left\{ \lim_{n\rightarrow\infty} \frac{1}{n}\sum_{i=1}^n g(X,Y_i) = f(X)\right\}$$ We want to show that $P[A]=1$. We have \begin{align} P[A] &= \sum_{x\in \mathcal{X}}P[A \cap \{X=x\}] \\ &\overset{(a)}{=} \sum_{x \in \mathcal{X}} P[A_x \cap \{X=x\}] \\ &\overset{(b)}{=} \sum_{x \in \mathcal{X}} P[X=x] \\ &= 1 \end{align} where (a) uses the fact that for each $x \in \mathcal{X}$ we have $$\{A \cap \{X=x\}\} = \{A_x \cap \{X=x\}\}$$ and (b) uses the fact that $P[A_x]=1$ and so $P[A_x\cap \{X=x\}] = P[X=x]$.$\Box$

• The more general case (with $X$ taking values in an uncountably infinite set) likely requires more measure theory tools than I can muster. I have asked a follow-up question here: math.stackexchange.com/questions/2756426/… Apr 27, 2018 at 16:30

Yes this is necessary take the function $g(x,y)=xy$, if instead of being independent $X,Y$ are completely dependent ie $X=Y$ we have that on the one hand $$\frac{1}{n}\sum_{i=1}^n g(X_1, Y_i)=X_1 E(X_1) \rightarrow_{a.s.} \text{ }\text{ as n\rightarrow \infty}$$ and on the other hand $$E_{Y}(g(X_1, Y_i))=E(X_1)^2 i\geq 2 .$$

• Thank you for your help. I'm not completely sure I understand your answer (see my edit above). Could you reformulate it by dividing in subparts following my questions?
– Star
Apr 26, 2018 at 13:29
• I do not understand your answer either. The notation is unusual: (i) What is meant by $X=Y$ in terms of $X_i$ and $Y_i$? (ii) What is meant by $E_Y[\cdot]$ when the expression inside the expectation has both $X_1$ and $Y_i$? This goes back to the Nate Eldredge comment on ambiguity of "expectation with respect to $Y_i$ only." It would likely be more clear to write expectations without subscripts, such as $E[Z]$ or $E[Z|W]$. Apr 26, 2018 at 14:28
• @CGT Ok, I think we are getting confused for me $X=Y$ means that $X$ and $Y$ are the same stochastic proceses (In this context it is equivalent to $X_i= Y_i$ for all $i$. Respect to the conditional expectation, for me that notation refears for the condtional expectation w.r.t. a sigma algebra, in this case the sigma algebra generated by Y. Apr 26, 2018 at 17:42