# An odd definition of a conditional expectation

I'm trying to understand why the author of a set of lecture notes on learning theory picked a particular definition (notes are here, page 10).

We have a set $$\mathcal{X}$$, a random element $$X$$ of $$\mathcal{X}$$ and a random variable $$Y \in \mathcal{Y}$$, $$\mathcal{Y} \subset \mathbb{R}$$. The author defines the regression function $$\eta \colon \mathcal{X} \to \mathcal{Y}$$ to be that satisfying $$\mathbb{E}[Y\phi(X)] = \mathbb{E}[\eta(X)\phi(X)]$$ for all bounded continuous functions $$\phi \colon \mathcal{X} \to \mathbb{R}$$.

As far as I can see, $$\eta$$ has to be almost surely equal to $$\mathbb{E}[Y|X]$$. Am I missing something? Why not just use the conditional expectation as the definition? The notes are excellent, so I figure there's a reason!

• I think that you are right that their definition characterizes the conditional expectation. I don't see a learning-theory reason why they would focus on this definition, but perhaps they see one? Maybe they feel that the standard definition of the conditional expectation is too confusing and they prefer this formulation? Commented Jul 22 at 15:02

## 2 Answers

Because the definition is equivalent to $$\mathbb{E}\left\lbrace\left[Y-\eta(X)\right] \phi(X) \right\rbrace = 0,$$ we can intuitively interpret that the residual $$Y-\eta(X)$$ from the regression is no longer co-varying with any $$\phi(X)$$ in terms of expectation. In this sense, $$\eta(X)$$ has explained variation in $$Y$$ to its best degree, i.e., $$\eta$$ has contained as much information of $$X$$ as possible that is useful to approximate $$Y$$.

Thus, such an $$\eta$$ is an optimal regression function in the expectation sense. Other regression functions can then be compared to this optimum. So this definition, compared to $$\mathbb{E}(Y\mid X)$$, better emphasizes the performance aspect of $$\eta$$.

It seems to be just a matter of terminology. Assuming $$Y$$ to be integrable, you have the conditional expectation $$Z:=\Bbb E[Y\,|\, X]$$. As $$Z$$ is $$\sigma (X)$$ measurable, there is a Borel function $$\eta$$ such that $$Z=\eta(X)$$, almost surely, as you observe. The label "regression function" betrays a stistical orientation.