5
$\begingroup$

I am trying to understand the sentence

the mean minimizes the mean squared error.

from wikipedia https://en.wikipedia.org/wiki/Average_absolute_deviation. From a previous post Formal proof that mean minimize squared error function I can see a formal proof which according to the authors prove that the mean minimaze. But my objection on that prove is that is you substitute m with median all the reasoning still stands.

Therefore, I restricted myself to one dimension and built the following example :

MSE = 1/n[SUM(Y_hat_i - Y_i)**2]

Given the following distribution Y = [5,3,2,7,4], minimizing MSE I intended as in the following: I should find a function Y_hat such that when it is applied to the MSE formula (more specifically to each element of the Y distribution), we are sure that we obtain the minimum value. For simplicity, I could consider the Y_hat as any measure or function for the central tendency, including, mean, median, mode, etc.

Lets suppose, Y_hat = mean (=4.2 for our distribution), therefore, I did all the calculations such as:

1/5 [(4.2-5)**2 + (4.2-3)**2+ (4.2-2)**2 + (4.2-7)**2  + (4.2-4)**2 ] = 2.96

Let's suppose, Y_hat = median (=4 for our distribution), therefore, I did all the calculations such as:

1/5 [(4-5)**2 + (4-3)**2+ (4-2)**2 + (4-7)**2  + (4-4)**2 ] = 2.8

If I did all the calculation correctly, the result shows that for this case, the median is minimizing better than the mean.

I am sure, I am doing something wrong and missing important aspects of the overall reasoning.

Please, could you provide any clarification on the above topic? I would really appreciate it.

Many Thanks in advance, Best Regards, Carlo

$\endgroup$
7
$\begingroup$

If you have $(y_i)_{i=1}^n$, consider the mean squared difference from the $y_i$ to a value $a$.

This is $s(a) =\sum_{i=1}^n (y_i-a)^2 $.

Manipulating this,

$\begin{array}\\ s(a) &=\sum_{i=1}^n (y_i-a)^2\\ &=\sum_{i=1}^n (y_i^2-2ay_i+a^2)\\ &=\sum_{i=1}^n y_i^2-\sum_{i=1}^n2ay_i+\sum_{i=1}^na^2\\ &=\sum_{i=1}^n y_i^2-2a\sum_{i=1}^ny_i+na^2\\ \end{array} $

There are a number of ways to minimize this expression. Perhaps the easiest is to differentiate with respect to $a$. This gives $s'(a) =-2\sum_{i=1}^ny_i+2na $ and this is zero when $a =\dfrac{\sum_{i=1}^ny_i}{n} $, the mean of the values.

Note that, since $s''(a) =2n > 0 $, this value of $a$ gives a minimum.


(added later)

An even easier way is to write $\bar{y} =\dfrac{\sum_{i=1}^ny_i}{n} $ and $\bar{y^2} =\dfrac{\sum_{i=1}^ny_i^2}{n} $.

Then

$\begin{array}\\ \frac1{n}s(a) &=\frac1{n}\sum_{i=1}^n y_i^2-2a\frac1{n}\sum_{i=1}^ny_i+a^2\\ &=a^2-2a\bar{y}+\bar{y^2}\\ &=a^2-2a\bar{y}+\bar{y}^2-\bar{y}^2+\bar{y^2}\\ &=(a-\bar{y})^2+\bar{y^2}-\bar{y}^2\\ \end{array} $

Since $\bar{y^2}-\bar{y}^2$ is independent of $a$, this is clearly a minimum when $a = \bar{y}$ and the value at the minimum is $\bar{y^2}-\bar{y}^2$.

$\endgroup$
  • $\begingroup$ Many Thanks MartyCohen. This is what I was looking for. Your prove clearly shows that to min MSE, Y_hat MUST be the mean. Thank you once again. $\endgroup$ – Carlo Allocca Dec 6 '17 at 19:09
3
$\begingroup$

Check your calculation, we have $$ \frac{(4-5)^2 + (4-3)^2+ (4-2)^2 + (4-7)^2 + (4-4)^2}{5} = \frac{15}{5} = 3 > 2.96. $$ The mean minimizes the MSE indeed, no contradiction here.

$\endgroup$
  • $\begingroup$ Yes, you are right Martin. Thanks for the double check. My question is Why the mean is the function that minimizes the MSE? $\endgroup$ – Carlo Allocca Dec 6 '17 at 18:57
  • $\begingroup$ I am looking at the prove given here math.stackexchange.com/questions/967138/… but if I substitute m with median everything still work. What am I missing? $\endgroup$ – Carlo Allocca Dec 6 '17 at 18:59
  • $\begingroup$ I am no statistician, but it is common in calculus for the absolute min value to occur at several different $x$s. Could the same thing be happening here? Or, in your case, are the mean and median equal so there is no contradiction? $\endgroup$ – Randall Dec 6 '17 at 19:05
  • $\begingroup$ Thanks Randall. Problem solved by MartyCohen above. $\endgroup$ – Carlo Allocca Dec 6 '17 at 19:10
  • $\begingroup$ @CarloAllocca I would say that the post of Reiner Martin matches more your question. $\endgroup$ – callculus Dec 6 '17 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.