Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The question is:

Assuming that $y_i = \mu + \epsilon_i $,$i = 1,\ldots,n$ with independent and identically distributed errors $\epsilon_i$ such that $E[\epsilon_i] = 0$ and $Var[\epsilon_i] = \sigma^2$, find the least squares estimator of $\mu$. Find its variance.

I'm not sure how to go about doing this.

I know that the least squares bit means that I minimize the sum of the errors, and so I would have to use the formula:

$$\sum_i (y_i - \mu)^2$$

and then differentiate (wrt to $\mu$?) and then let it equal 0.

Is that correct?

Once I've done this, I would I calculate its $E[\mu]$, because I don't have any definition for $\mu$. Or is $\mu = \beta_0 + \beta_1 \cdot x_i$? If it is, then isn't the estimator the same?

share|cite|improve this question
$\mu = \beta_0 + \beta_1 \cdot x_i$ would be a wrong approach: the question does not mention any $x_i$. – Henry Sep 29 '12 at 21:09
up vote 1 down vote accepted

Some hints, but not quite the full answer:

There is a difference between a parameter $\mu$ and an estimator of that parameter. So if we call the estimator $\hat{\mu}$ then you want to minimise $$\sum_i (y_i - \hat{\mu})^2$$ which is $$\sum_i y_i^2 - \sum_i 2 y_i \hat{\mu} +\sum_i \hat{\mu} ^2$$ and (as you suggest) this will be when its derivative with respect to $\hat{\mu}$ is zero. Strictly speaking you should check this is a minimum, but since the derivative is monotone increasing that is obvious.

Since $y_i = \mu + \epsilon_i$, you know $E[y_i] = E[\mu] + E[\epsilon_i]$, so it will be easy to find $E[\hat{\mu}]$.

As for $Var(\hat{\mu})$, you again have to multiply out a square, looking at $$E\left[\left(\hat{\mu}-E[\hat{\mu}]\right)^2\right].$$ You might want to use the fact that $y_i^2 = \mu^2 + 2 \mu \epsilon_i +\epsilon_i^2$ implies $E[y_i^2] = \mu^2 + \sigma^2$.

share|cite|improve this answer
Nice hints, solution is right behind the corner. – Roman Luštrik Sep 29 '12 at 21:20

$$ \begin{align} \sum_{i=1}^n (y_i-\mu)^2 & = \sum_{i=1}^n ((y_i - \bar y) + (\bar y -\mu))^2 \\[8pt] & = \sum_{i=1}^n \left((y_i-\bar y)^2 + 2(y_i-\bar y)(\bar y-\mu) + (\bar y -\mu)^2 \right) \\[8pt] & = \left(\sum_{i=1}^n (y_i-\bar y)^2 \right) + \left(2(\bar y -\mu)\sum_{i=1}^n (y_i-\bar y)\right) + n(\bar y -\mu)^2 \\[8pt] & = (\text{something not depending on }\mu) + (0) + n(\bar y -\mu)^2. \end{align} $$ (The second sum is $0$: if you subtract the average of the numbers in a list from each of the numbers, then the sum of the resulting numbers is $0$.) So you're looking for the value of $\mu$ that minimizes $(\bar y - \mu)^2$, and that's obviously $\mu=\bar y$.

None of this relies on the fact that the errors are normally distributed.

Next question: variance of $\bar y$: $$ \operatorname{var}(\bar y) = \operatorname{var}\left(\frac{y_1+\cdots+y_n}{n}\right) = \frac{1}{n^2}\operatorname{var}(y_1+\cdots+y_n) $$ $$ = \frac{1}{n}(\sigma^2 + \cdots+\sigma^2) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}. $$ Here we relied on independence to conclude that the sum of the variances equals the variance of the sum, but we still didn't need the fact that the errors are normally distributed.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.