In Bishop's book [1] they show that the optimal y(x) w.r.t. squared error loss function

$$E[L]=\int \int \{y(x)-t\}^2p(x,t)dxdt $$

is given by a conditional expectation $y(x) = E_t[t|x]$. However, in the derivation of this result, they just say "using the calculus of variations we have":

$$ \frac{\delta E[L]}{\delta y(x)} = 2 \int \{y(x)-t\}p(x,t)dt $$

How to use calculus of variations to come to this result? Are there any simple rules to use like for the ordinary differentiation wrt to variables?


[1] Pattern Recognition and Machine Learning (p. 46)


1 Answer 1


The book mentioned that $y(x)$ is "completely flexible", hence I suppose $y(x)$ is a minimizer of the functional over all continuous functions. Let $\phi(x)$ be a continuous test function, then $y(x)+s\phi(x)$ is also in the admissible set (that is a continuous function in this case).

Hence $g(s)=\int \int \{y(x)+s\phi(x)-t\}^2p(x,t)dxdt $ obtains local minimum at $s=0$. Differentiate w.r.t $s$,

$$g'(s)=\int \int 2\{y(x)+s\phi(x)-t\}p(x,t)\phi(x)dxdt=$$ $$=\int \int 2\{y(x)+s\phi(x)-t\}p(x,t)dt\phi(x)dx$$

Hence $$0=g'(0)=\int \int 2\{y(x)-t\}p(x,t)dt\phi(x)dx$$ for all continous function $\phi(x)$.

Thus we have $\int 2\{y(x)-t\}p(x,t)dt=0$.

  • $\begingroup$ Thanks a lot! Just a clarification of the last step. It must be that $\int 2\{y(x)-t\}p(x,t)dt=0$ because $\phi(x)$ can be any function, even $\phi(x)$. Hence there is no other way for $\int \int 2\{y(x)-t\}p(x,t)dt\phi(x)dx$ being zero than that $\int 2\{y(x)-t\}p(x,t)dt=0$ is zero. $\endgroup$
    – ticcky
    Commented May 6, 2015 at 12:19
  • $\begingroup$ @ticcky Basically you are right. One thing you need to notice is that $\int 2\{y(x)-t\}p(x,t)dt$ is a continuous function of $x$, and continuous $f$ satisfying $\int f^2(x) dx=0$ implies $f=0$ $\endgroup$
    – John
    Commented May 6, 2015 at 15:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .