Proving that the estimate of a mean is a least squares estimator [duplicate]

This question is an exact duplicate of:

I think this is a really simple question so please bear with me -- I just had my first class in regression and I'm a little confused about nomenclature/labeling.

Does anyone recommend some good weblinks that explain beginning linear regression really well?

There's a question I've been looking at for a while and I'm not sure how to do it (although I'm sure the solution is simple):

Show that the sample estimate $\hat{\mu} = \bar X = \frac{1}{n} \sum X_i$ is a least square estimator of $\mu$ for a variable $X$ given $X_1, \ldots, X_n$.

My first thought was, $$SSE = \sum (\mu - \hat{\mu})^2,$$

but I'm not sure if that's right. I'm confused about what the $\beta$ is (is it $n$?) and I don't know if there are enough parameters to expand it.

Thanks so much for your patience and if this doesn't make sense, I can clarify more. Thanks!

marked as duplicate by Rahul, egreg, Hagen von Eitzen, Stefan Hansen, Lost1Feb 3 '14 at 18:18

This question was marked as an exact duplicate of an existing question.

migrated from stackoverflow.comFeb 1 '14 at 18:55

This question came from our site for professional and enthusiast programmers.

Let's say that you have $n$ independent and identically distributed (real valued) random variables $(X_{1},\ldots,X_{n})$ with distribution $\mathcal{N}(\mu,\sigma^{2})$. Here, $(\mu,\sigma^{2})$ are unknown parameters which we can estimate. Note that we assume that all random varialbes $X_{i}$ have the same mean $\mu$. In other words, we could say that we assume the following model :

$$\forall i \in \left\{1,\ldots,n \right\}, \; X_{i} = \mu + \varepsilon_{i} \tag{\star}$$

where $(\varepsilon_{1},\ldots,\varepsilon_{n})$ are independent and identically distributed random variables with distribution $\mathcal{N}(0,\sigma^{2})$. One way to estimate the mean $\mu$ is to consider its least squares estimate $\hat{\mu}_{n}$, where $\hat{\mu}_{n}$ is defined as follows :

$$\hat{\mu}_{n} = \mathop{\mathrm{argmin}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2}$$

To determine this estimate, you want to minimize the function $f$ given by :

$$f(\mu) = \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2} = \sum_{i=1}^{n} X_{i}^{2} - 2\mu \sum_{i=1}^{n} X_{i} + n\mu^{2}.$$ $f$ is a differentiable function whose derivative is :

$$f'(\mu) = -2 \sum_{i=1}^{n} X_{i} + 2n \mu$$

So, $f'(\mu) = 0$ if and only if $\displaystyle \mu = \frac{1}{n} \sum_{i=1}^{n} X_{i}$. Since $\displaystyle \lim \limits_{\mu \to \pm \infty} f(\mu) = +\infty$, the critical point of $f'$ we found is actually the global minimum of $f$. As a conclusion :

$$\boxed\hat{\mu}_{n} = \frac{1}{n} \sum_{i=1}^{n} X_{i}}$$

Minimizing a sum of squared "errors" is not the only way to determine the parameters in $(\star)$. The, perhaps, most common way is to determine maximum likelihood estimates. This method consists in maximizing a function (the likelihood function), which, in the case of the model $(\star)$, writes :

$$\ell \big( \mu,\sigma \big) = \frac{1}{\big( \sigma \sqrt{2\pi} \big)^{n}} \exp \Big( - \frac{1}{2\sigma^{2}} \sum_{i=1}^{n} (x_{i} - \mu)^{2} \Big)$$

Instead of maximizing $\ell$, it is more convenient to maximize $\log \ell$, the log-likelihood. Maximizing $\log \ell$ with respect to $\mu$ (consider $\sigma$ fixed) yields :

\begin{align*} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \log \ell & = {} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i}-\mu \big)^{2} \\ &= \hat{\mu}_{n} \\ \end{align*}

So, our least squares estimates is also (in this case) the maximum likelihood estimate of the mean.