12
$\begingroup$

I am trying to understand the sentence

the mean minimizes the mean squared error.

from wikipedia https://en.wikipedia.org/wiki/Average_absolute_deviation. From a previous post Formal proof that mean minimize squared error function I can see a formal proof which according to the authors prove that the mean minimaze. But my objection on that prove is that is you substitute m with median all the reasoning still stands.

Therefore, I restricted myself to one dimension and built the following example :

MSE = 1/n[SUM(Y_hat_i - Y_i)**2]

Given the following distribution Y = [5,3,2,7,4], minimizing MSE I intended as in the following: I should find a function Y_hat such that when it is applied to the MSE formula (more specifically to each element of the Y distribution), we are sure that we obtain the minimum value. For simplicity, I could consider the Y_hat as any measure or function for the central tendency, including, mean, median, mode, etc.

Lets suppose, Y_hat = mean (=4.2 for our distribution), therefore, I did all the calculations such as:

1/5 [(4.2-5)**2 + (4.2-3)**2+ (4.2-2)**2 + (4.2-7)**2  + (4.2-4)**2 ] = 2.96

Let's suppose, Y_hat = median (=4 for our distribution), therefore, I did all the calculations such as:

1/5 [(4-5)**2 + (4-3)**2+ (4-2)**2 + (4-7)**2  + (4-4)**2 ] = 2.8

If I did all the calculation correctly, the result shows that for this case, the median is minimizing better than the mean.

I am sure, I am doing something wrong and missing important aspects of the overall reasoning.

Please, could you provide any clarification on the above topic? I would really appreciate it.

Many Thanks in advance, Best Regards, Carlo

$\endgroup$

3 Answers 3

18
$\begingroup$

If you have $(y_i)_{i=1}^n$, consider the mean squared difference from the $y_i$ to a value $a$.

This is $s(a) =\sum_{i=1}^n (y_i-a)^2 $.

Manipulating this,

$\begin{array}\\ s(a) &=\sum_{i=1}^n (y_i-a)^2\\ &=\sum_{i=1}^n (y_i^2-2ay_i+a^2)\\ &=\sum_{i=1}^n y_i^2-\sum_{i=1}^n2ay_i+\sum_{i=1}^na^2\\ &=\sum_{i=1}^n y_i^2-2a\sum_{i=1}^ny_i+na^2\\ \end{array} $

There are a number of ways to minimize this expression. Perhaps the easiest is to differentiate with respect to $a$. This gives $s'(a) =-2\sum_{i=1}^ny_i+2na $ and this is zero when $a =\dfrac{\sum_{i=1}^ny_i}{n} $, the mean of the values.

Note that, since $s''(a) =2n > 0 $, this value of $a$ gives a minimum.


(added later)

An even easier way is to write $\bar{y} =\dfrac{\sum_{i=1}^ny_i}{n} $ and $\bar{y^2} =\dfrac{\sum_{i=1}^ny_i^2}{n} $.

Then

$\begin{array}\\ \frac1{n}s(a) &=\frac1{n}\sum_{i=1}^n y_i^2-2a\frac1{n}\sum_{i=1}^ny_i+a^2\\ &=a^2-2a\bar{y}+\bar{y^2}\\ &=a^2-2a\bar{y}+\bar{y}^2-\bar{y}^2+\bar{y^2}\\ &=(a-\bar{y})^2+\bar{y^2}-\bar{y}^2\\ \end{array} $

Since $\bar{y^2}-\bar{y}^2$ is independent of $a$, this is clearly a minimum when $a = \bar{y}$ and the value at the minimum is $\bar{y^2}-\bar{y}^2$.

$\endgroup$
1
  • $\begingroup$ Many Thanks MartyCohen. This is what I was looking for. Your prove clearly shows that to min MSE, Y_hat MUST be the mean. Thank you once again. $\endgroup$ Commented Dec 6, 2017 at 19:09
4
$\begingroup$

Check your calculation, we have $$ \frac{(4-5)^2 + (4-3)^2+ (4-2)^2 + (4-7)^2 + (4-4)^2}{5} = \frac{15}{5} = 3 > 2.96. $$ The mean minimizes the MSE indeed, no contradiction here.

$\endgroup$
6
  • $\begingroup$ Yes, you are right Martin. Thanks for the double check. My question is Why the mean is the function that minimizes the MSE? $\endgroup$ Commented Dec 6, 2017 at 18:57
  • $\begingroup$ I am looking at the prove given here math.stackexchange.com/questions/967138/… but if I substitute m with median everything still work. What am I missing? $\endgroup$ Commented Dec 6, 2017 at 18:59
  • $\begingroup$ I am no statistician, but it is common in calculus for the absolute min value to occur at several different $x$s. Could the same thing be happening here? Or, in your case, are the mean and median equal so there is no contradiction? $\endgroup$
    – Randall
    Commented Dec 6, 2017 at 19:05
  • $\begingroup$ Thanks Randall. Problem solved by MartyCohen above. $\endgroup$ Commented Dec 6, 2017 at 19:10
  • $\begingroup$ @CarloAllocca I would say that the post of Reiner Martin matches more your question. $\endgroup$ Commented Dec 6, 2017 at 19:13
1
$\begingroup$

More formally and generally we may state that expected value minimizes mean square error.

I.e. $c=\mathbb{E}(X)$ minimizes $\mathbb{E}((X-c)^2)$.

See two related cross-validated questions:

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .