Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I want to find a function $f$ that given $x$ will predict $y$. The expected prediction error of $f$ is $$e = E[(Y-f(X))^2]=\int \int [y-f(x)]^2 p(x,y) dx dy$$ the expectation of $(Y-f(X))^2$ with respect to the joint distribution $p(X,Y)$. $X\in\mathbb R$, $Y\in\mathbb R$.

Others told me the optimal function that minimises $e$ is $$f(x) = E[Y|X=x]$$ the conditional expectation of $Y$ given $X=x$. Is that a right suggestion? This may be a simple proof to many people, but I don't know how to show/proof it, can someone help me?

share|improve this question

1 Answer 1

up vote 3 down vote accepted

Let $f(x) = \mathbb{E}[Y|X=x]$. Let $g(x)$ be another function. We shall now prove that $$\mathbb{E}[\left(Y-g(X)\right)^2] \geq \mathbb{E}[\left(Y-f(X)\right)^2]$$

$$\mathbb{E}[\left(Y-g(X)\right)^2] = \mathbb{E}[\left(Y-f(X) + f(X) - g(X)\right)^2]$$

$$\mathbb{E}[\left(Y-g(X)\right)^2] = \mathbb{E}[\left(Y-f(X) \right)^2] + 2 \mathbb{E}[\left(Y-f(X) \right)\left(f(X) - g(X)\right)] + \mathbb{E}[\left(f(X) - g(X)\right)^2]$$

Now note that $$\mathbb{E}[\left(Y-f(X) \right)h(X)] = 0$$ $\forall h(x) \in L^2$

The above can be obtained from the tower property or Law of total expectation as follows $$\mathbb{E}[\left(Y-f(X) \right)h(X)] = \mathbb{E}[\mathbb{E}[\left(Y-f(X) \right)h(X) |X]] = \mathbb{E}[\mathbb{E}[\left(Y-f(X)|X] \right)h(X) ] = 0$$

Hence, we get $$\mathbb{E}[\left(Y-g(X)\right)^2] = \mathbb{E}[\left(Y-f(X) \right)^2] + \mathbb{E}[\left(f(X) - g(X)\right)^2]$$

Hence, $$\mathbb{E}[\left(Y-g(X)\right)^2] \geq \mathbb{E}[\left(Y-f(X) \right)^2]$$

The above should suggest some sort of generalized Pythagoras theorem and this is true from the following generalized statement.

A generalized version of the above is, If we have a closed convex subset $A$ of a Hilbert space $\mathscr{H}$, then $\forall h \in \mathscr{H}$, $!\exists a \in A$ such that $||h-a|| < ||h-y||$, $\forall y \in A \backslash \{a\}$. Further if $A$ happens to be a closed subspace, then the element $a$ is the "projection" of $h$ onto the space $A$.

The proof for this version is identical to the proof written above.

share|improve this answer
    
Do we need to use the $p(x,y)$ in the proof? I know the fact that $p(x,y)=p(y|x)p(x)$, and can this be used? –  Dean Jun 4 '11 at 23:46
    
@Dean: You do not need to use $p(x,y) = p(y|x) p(x)$ explicitly. All you need for this proof is to use the tower property or the law of total expectation en.wikipedia.org/wiki/Law_of_total_expectation –  user17762 Jun 5 '11 at 3:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.