14
$\begingroup$

I know that the variance measures the dispersion of an estimator around its mean i.e. $\sigma^2 = E[(X - \mu)^2]$ (the second central moment about the mean).

But I'm not getting the meaning of the definition below:

The mean squared error measures the dispersion around the true value of the parameter being estimated. If the estimator is unbiased then both are identical.

I know that both variance and MSE are related to second moment. But I'm not getting the actual difference between them. Can anybody explain to me the basic difference between them in simple language?

$\endgroup$
6
$\begingroup$

The variance measures how far a set of numbers is spread out whereas the MSE measures the average of the squares of the "errors", that is, the difference between the estimator and what is estimated. The MSE of an estimator $\hat{\theta}$ of an unknown parameter $\theta$ is defined as $E[(\hat{\theta}-\theta)^2]$.

The MSE is the second moment (about the origin) of the error, that's why it includes both the variance of the estimator and its bias (the bias being $E(\hat{\theta})-\theta$).

In other words, the variance just measures the dispersion of the values; the MSE indicates how different the values of the estimator and the actual values of the parameters are. The MSE is a comparison of the estimator and the true parameter, as it were. That's the difference.

Edit: I'll use your example: Suppose we have a bull's-eye, the mean of the estimator is the target. The variance measures how far the arrows are from the target. Now suppose we have another bull's-eye, and this time the target is the true parameter. The MSE measures how far the arrows (estimates) are from the target. In the first case, we just measure the dispersion of the values of the estimator with respect to its mean. In the second case, we measure the error we make when estimating the parameter, i.e. we are comparing it with the true parameter (that's why we want estimators with variance and MSE as small as possible). Don't confuse the mean of an estimator with the true value of a parameter.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Can you explain it with the help of any example. Suppose the parameter is the bull's-eye of a target, the estimator is the process of shooting arrows at the target, and the individual arrows are estimates (samples). On the basis of above example can you explain Variance and MSE. $\endgroup$ – Atinesh Jul 12 '15 at 10:33
  • $\begingroup$ @Atinesh I've edited my answer and I've used your example. Let me know if any part of my explanation needs clarification. $\endgroup$ – Cristopher Jul 12 '15 at 16:59
  • $\begingroup$ Please see the below Image. Is my understanding correct. s27.postimg.org/el70ruqeb/image.png $\endgroup$ – Atinesh Jul 13 '15 at 7:14
  • $\begingroup$ @Atinesh It's not clear to me what you tried to draw there. Re-read the last sentence in my answer. I think you are still a bit confused. See this picture, it illustrates the difference between variance and MSE very well: i.imgur.com/p4VSiAZ.png $\endgroup$ – Cristopher Jul 13 '15 at 14:52
  • $\begingroup$ I saw your above image. I tried to say the same thing with 2-D space, that you explained with 1D Space. In the image above cross points represents True Value/ Actual Value and the dot points represents Estimated Value. See the below upated image s1.postimg.org/5qbk5gwr3/image.png $\endgroup$ – Atinesh Jul 13 '15 at 15:26
16
$\begingroup$

The main difference is whether you are considering the deviation of the estimator of interest from the true parameter (this is the mean squared error), or the deviation of the estimator from its expected value (this is the variance). Consequently, we can see that when the bias of the estimator is zero, the variance and mean squared error are equal.

Mathematically, if $\hat \theta$ is an estimator for $\theta$, then $$\operatorname{MSE}[\hat \theta] = \operatorname{E}[(\hat\theta - \theta)^2],$$ whereas $$\operatorname{Var}[\hat\theta] = \operatorname{E}[(\hat\theta - \operatorname{E}[\hat\theta])^2].$$ And regarding the previous remark, $$\operatorname{Bias}[\hat\theta] = \operatorname{E}[\hat\theta - \theta],$$ so when the bias is zero, $\operatorname{E}[\hat\theta] = \theta$ and now we easily see how the MSE and variance become equivalent.

Note, however, we can also write: $$\operatorname{Var}[\hat\theta] = \operatorname{E}[\hat \theta^2 - 2\hat\theta \operatorname{E}[\hat \theta] + \operatorname{E}[\hat\theta]^2] = \operatorname{E}[\hat\theta^2] - \operatorname{E}[\hat\theta]^2,$$ so that $$\begin{align*} \operatorname{Var}[\hat\theta] + \operatorname{Bias}^2[\hat\theta] &= \operatorname{E}[\hat\theta^2] - \operatorname{E}[\hat\theta]^2 + (\operatorname{E}[\hat \theta] - \theta)^2 \\ &= \operatorname{E}[\hat\theta^2] - 2\theta \operatorname{E}[\hat\theta] + \theta^2 \\ &= \operatorname{E}[(\hat \theta - \theta)^2] \\ &= \operatorname{MSE}[\hat\theta]. \end{align*}$$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.