3
$\begingroup$

I am trying to find the estimator of the variance $\sigma^2$ of a normal distribution with the minimum mean square error. From reading up, I know the unbiased estimator of the variance of a Guassian is $\frac{1}{n-1}\sum_{i=1}^{n}(X_i - \bar{X})^2$ and that the estimator I am looking for is a scaled version of this unbiased estimator. The question is from a problem sheet from the last academic year.

The specific question is:

Let $X_1, \dots, X_n$ be a simple random sample from a normal distribution with unknown mean $\mu$ and variance $\sigma^2$. Consider estimators of $\sigma^2$ of the form $k \sum_{i=1}^{n}(X_i - \bar{X})^2$ and find the value of $k$ that minimises the mean square error. What is the efficiency of the usual unbiased estimator relative to this estimator, if the relative efficiency is defined as the ratio of the mean squared error?

For the first part, I think I am meant to rewrite the MSE of the estimator as an expectation and then take derivatives with respect to $k$. This is what I have so far:

$$ \begin{align} MSE(\hat{\theta}) &= \mathbb{E} \left[ (\hat{\theta} - \theta)^2 \right] \\ &= \mathbb{E} \left[ \left(k \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X})^2 - \sigma^2 \right)^2 \right] \\ &= \mathbb{E} \left[ \left(k \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X} )^2 \right)^2 - 2 \left( k \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X})^2 \right) \sigma^2 + \sigma^4 \right] \end{align} $$

But however else I continue from here, I can't find a way that gets me to $ k = \frac{1}{n + 1}$, which the Wikipedia article linked to below suggests is the answer.

For the second part, I think I can use the MSE of the unbiased estimator given in the Wikipedia article to find the efficiency, although it would be really helpful to see the steps that one takes to calculate this MSE, as in the article it is just stated.

My question is linked to this one, although less advanced.

The Wikipedia article on the MSE linked to in the question above is also relevant, although there they also calculate $ \mathbb{E} [S^4_{n-1}]$, which I'm not sure about.

$\endgroup$
  • $\begingroup$ I doubt you can do this just with formal manipulations of the expectations -- it's a question specifically about the normal distribution, and you'll probably have to actually compute the expectation values. You have a quadratic function of $k$, so you can readily differentiate it and solve for $k$ -- but the result won't simplify unless you compute the expectations. $\endgroup$ – joriki Jul 23 '18 at 12:36
  • $\begingroup$ I got $k=\frac{1}{\color{red}{n-1}}$, when I minimized the MSE. $\endgroup$ – callculus Jul 23 '18 at 12:52
  • $\begingroup$ If you still need help for the first question give a reply. $\endgroup$ – callculus Jul 23 '18 at 13:18
  • $\begingroup$ I would have thought you wanted to minimise $\mathbb{E} \left[ \left(k \sum_{i=1}^{n}(X_i - \bar{X})^2 - \sigma^2 \right)^2 \right]$ rather than $\mathbb{E} \left[ \left(k \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X})^2 - \sigma^2 \right)^2 \right]$ $\endgroup$ – Henry Jul 23 '18 at 13:42
  • $\begingroup$ What you can do is show that the estimator $T_1=\frac{1}{n+1}\sum (X_i-\bar X)^2$ has lower MSE than the estimator $T_2=\frac{1}{n-1}\sum (X_i-\bar X)^2$. The former is biased for $\sigma^2$, while the latter is of course unbiased. $\endgroup$ – StubbornAtom Jul 23 '18 at 13:47
2
$\begingroup$

Assuming $s^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2$ is the sample variance, the only result one needs to use is that $$\frac{(n-1)s^2}{\sigma^2}\sim\chi^2_{n-1}$$

Keeping in mind that for a chi-squared random variable $X$ with $n$ degrees of freedom, mean and variance of $X$ is $n$ and $2n$ respectively, we have

\begin{align} \operatorname{Var}\left(\frac{(n-1)s^2}{\sigma^2}\right)&=2(n-1) \\\text{ or },\quad \operatorname{Var}(s^2)&=\frac{2\sigma^4}{n-1} \end{align}

We also find that $$E(s^2)=\sigma^2$$

What you are looking for is the MSE of an estimator of the form $cs^2$, $(c\ne 1)$.

Now,

\begin{align} \text{MSE}_{\sigma}(cs^2)&=\operatorname{Var}_{\sigma}(cs^2)+\left\{\text{bias}(cs^2)\right\}^2 \\&=c^2\frac{2\sigma^4}{n-1}+\left(c\sigma^2-\sigma^2\right)^2 \\&=\sigma^4\left[\frac{2c^2}{n-1}+(c-1)^2\right] \\&=\sigma^4\,\psi(c),\text{ say} \end{align}

Minimising $\psi(c)$ by usual calculus, we find that $c=\frac{n-1}{n+1}$ is the point of minima.

That means the estimator of the form $cs^2$ with the minimum MSE is $$T=\frac{1}{n+1}\sum_{i=1}^n(X_i-\bar X)^2$$

with the minimum MSE being $$\sigma^4\,\psi\left(\frac{n-1}{n+1}\right)=\frac{2\sigma^4}{n+1}$$

Of course the MSE of the estimator $s^2$ is its variance: $$\text{MSE}_{\sigma}(s^2)=\frac{2\sigma^4}{n-1}$$

So though $s^2$ is unbiased for $\sigma^2$, its mean squared error is more than that of the estimator $T$. While $T$ of course is biased for $\sigma^2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.