# Clever Solutions: Find $\min_{a,b} E[(V -(aU+b))^2]$ without differentiation?

Let $$(U,V)$$ be joint random variables (assume zero means for simplicity). It is well known that \begin{align} \min_{a,b} E[(V -(aU+b))^2], \end{align} is minimized by $$a= \frac{E[VU]}{E[U^2]}$$ and $$b=0$$. This answer results in what is know as the best linear estimator.

The classical proof of this expands the expression and finds partial derivatives with respect to $$a$$ and $$b$$.

My question: Are there alternative proofs that do not use differentiation? For example, can this be found via some known inequalities like Jensen or Cauchy-Schwarz? In other words, I am looking for interesting and unique solutions to this old problem.

Edit I didn't ask this originality, but can it be also done with minimal expansion.

## 2 Answers

Expanding, \begin{align*} \mathbb{E}(V-(aU+b))^2 &= \mathbb{E}V^2 - 2 (a \mathbb{E} UV + b \mathbb{E} V) + \mathbb{E}(aU+b)^2 \\ &= (a^2 \mathbb{E}U^2 - 2a \mathbb{E} UV) + b^2 + \mathbb{E}V^2 \end{align*} Note that these are separable quadratics in $$a, b$$ (since the cross term $$ab$$ vanishes because we assumed $$\mathbb{E} U = 0$$), and each is minimized at the vertex, given by $$x = -\gamma_1/2\gamma_2$$ in $$\gamma_2 x^2 + \gamma_1 x + \gamma_0 = 0$$. Hence, \begin{align*} a &= \frac{2 \mathbb{E} UV}{2 \mathbb{E}U^2} = \frac{\mathbb{E} UV}{\mathbb{E}U^2} \\ b &= -\frac{0}{2\cdot 1} = 0 \end{align*}

• Why do you assume that $E(U)=0$? – callculus Feb 15 at 18:28
• @callculus "assume zero means for simplicity" – Clement C. Feb 15 at 18:28
• Ah ok. I´ve didn´t noticed it. (+1) – callculus Feb 15 at 18:40

If you consider $$E(XY)$$ as scalar product of random variables, then your have the following meaning:

• $$E(X^2)$$ — is the squared length
• $$E((X-Y)^2)$$ — is the squared distance between $$X$$ and $$Y$$

Using this interpretation your problem is to find the random variable $$aU+b$$ that is the closest one to the random variable $$V$$. But all random variables of the form $$aU+b$$ are linear combinations of the random variable $$U$$ and constant $$1$$. So you just need to find the projection of random variable $$V$$ on this plane.

The difference $$V - (aU+b)$$ should be orthogonal to $$U$$ and $$1$$. The corresponding scalar product should be zero. So you have two equations $$\begin{cases} \langle V - (aU+b), 1 \rangle = E(V - (aU+b))=0 \\ \langle V - (aU+b), U \rangle = E((V - (aU+b))\cdot U)=0 \\ \end{cases}$$

From this system your get $$a$$ and $$b$$ :)