# Least squares estimator of mu

The question is:

Assuming that $y_i = \mu + \epsilon_i$,$i = 1,\ldots,n$ with independent and identically distributed errors $\epsilon_i$ such that $E[\epsilon_i] = 0$ and $Var[\epsilon_i] = \sigma^2$, find the least squares estimator of $\mu$. Find its variance.

I'm not sure how to go about doing this.

I know that the least squares bit means that I minimize the sum of the errors, and so I would have to use the formula:

$$\sum_i (y_i - \mu)^2$$

and then differentiate (wrt to $\mu$?) and then let it equal 0.

Is that correct?

Once I've done this, I would I calculate its $E[\mu]$, because I don't have any definition for $\mu$. Or is $\mu = \beta_0 + \beta_1 \cdot x_i$? If it is, then isn't the estimator the same?

-
$\mu = \beta_0 + \beta_1 \cdot x_i$ would be a wrong approach: the question does not mention any $x_i$. – Henry Sep 29 '12 at 21:09

Some hints, but not quite the full answer:

There is a difference between a parameter $\mu$ and an estimator of that parameter. So if we call the estimator $\hat{\mu}$ then you want to minimise $$\sum_i (y_i - \hat{\mu})^2$$ which is $$\sum_i y_i^2 - \sum_i 2 y_i \hat{\mu} +\sum_i \hat{\mu} ^2$$ and (as you suggest) this will be when its derivative with respect to $\hat{\mu}$ is zero. Strictly speaking you should check this is a minimum, but since the derivative is monotone increasing that is obvious.

Since $y_i = \mu + \epsilon_i$, you know $E[y_i] = E[\mu] + E[\epsilon_i]$, so it will be easy to find $E[\hat{\mu}]$.

As for $Var(\hat{\mu})$, you again have to multiply out a square, looking at $$E\left[\left(\hat{\mu}-E[\hat{\mu}]\right)^2\right].$$ You might want to use the fact that $y_i^2 = \mu^2 + 2 \mu \epsilon_i +\epsilon_i^2$ implies $E[y_i^2] = \mu^2 + \sigma^2$.

-
Nice hints, solution is right behind the corner. – Roman Luštrik Sep 29 '12 at 21:20

\begin{align} \sum_{i=1}^n (y_i-\mu)^2 & = \sum_{i=1}^n ((y_i - \bar y) + (\bar y -\mu))^2 \\[8pt] & = \sum_{i=1}^n \left((y_i-\bar y)^2 + 2(y_i-\bar y)(\bar y-\mu) + (\bar y -\mu)^2 \right) \\[8pt] & = \left(\sum_{i=1}^n (y_i-\bar y)^2 \right) + \left(2(\bar y -\mu)\sum_{i=1}^n (y_i-\bar y)\right) + n(\bar y -\mu)^2 \\[8pt] & = (\text{something not depending on }\mu) + (0) + n(\bar y -\mu)^2. \end{align} (The second sum is $0$: if you subtract the average of the numbers in a list from each of the numbers, then the sum of the resulting numbers is $0$.) So you're looking for the value of $\mu$ that minimizes $(\bar y - \mu)^2$, and that's obviously $\mu=\bar y$.

None of this relies on the fact that the errors are normally distributed.

Next question: variance of $\bar y$: $$\operatorname{var}(\bar y) = \operatorname{var}\left(\frac{y_1+\cdots+y_n}{n}\right) = \frac{1}{n^2}\operatorname{var}(y_1+\cdots+y_n)$$ $$= \frac{1}{n}(\sigma^2 + \cdots+\sigma^2) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}.$$ Here we relied on independence to conclude that the sum of the variances equals the variance of the sum, but we still didn't need the fact that the errors are normally distributed.

-