This question is an exact duplicate of:

I think this is a really simple question so please bear with me -- I just had my first class in regression and I'm a little confused about nomenclature/labeling.

Does anyone recommend some good weblinks that explain beginning linear regression really well?

There's a question I've been looking at for a while and I'm not sure how to do it (although I'm sure the solution is simple):

Show that the sample estimate $\hat{\mu} = \bar X = \frac{1}{n} \sum X_i$ is a least square estimator of $\mu$ for a variable $X$ given $X_1, \ldots, X_n$.

My first thought was, $$SSE = \sum (\mu - \hat{\mu})^2,$$

but I'm not sure if that's right. I'm confused about what the $\beta$ is (is it $n$?) and I don't know if there are enough parameters to expand it.

Thanks so much for your patience and if this doesn't make sense, I can clarify more. Thanks!


marked as duplicate by Rahul, egreg, Hagen von Eitzen, Stefan Hansen, Lost1 Feb 3 '14 at 18:18

This question was marked as an exact duplicate of an existing question.

migrated from stackoverflow.com Feb 1 '14 at 18:55

This question came from our site for professional and enthusiast programmers.


Let's say that you have $n$ independent and identically distributed (real valued) random variables $(X_{1},\ldots,X_{n})$ with distribution $\mathcal{N}(\mu,\sigma^{2})$. Here, $(\mu,\sigma^{2})$ are unknown parameters which we can estimate. Note that we assume that all random varialbes $X_{i}$ have the same mean $\mu$. In other words, we could say that we assume the following model :

$$ \forall i \in \left\{1,\ldots,n \right\}, \; X_{i} = \mu + \varepsilon_{i} \tag{$\star$}$$

where $(\varepsilon_{1},\ldots,\varepsilon_{n})$ are independent and identically distributed random variables with distribution $\mathcal{N}(0,\sigma^{2})$. One way to estimate the mean $\mu$ is to consider its least squares estimate $\hat{\mu}_{n}$, where $\hat{\mu}_{n}$ is defined as follows :

$$ \hat{\mu}_{n} = \mathop{\mathrm{argmin}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2} $$

To determine this estimate, you want to minimize the function $f$ given by :

$$ f(\mu) = \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2} = \sum_{i=1}^{n} X_{i}^{2} - 2\mu \sum_{i=1}^{n} X_{i} + n\mu^{2}.$$ $f$ is a differentiable function whose derivative is :

$$ f'(\mu) = -2 \sum_{i=1}^{n} X_{i} + 2n \mu $$

So, $f'(\mu) = 0$ if and only if $\displaystyle \mu = \frac{1}{n} \sum_{i=1}^{n} X_{i}$. Since $\displaystyle \lim \limits_{\mu \to \pm \infty} f(\mu) = +\infty$, the critical point of $f'$ we found is actually the global minimum of $f$. As a conclusion :

$$ \boxed{\displaystyle \hat{\mu}_{n} = \frac{1}{n} \sum_{i=1}^{n} X_{i}} $$

Minimizing a sum of squared "errors" is not the only way to determine the parameters in $(\star)$. The, perhaps, most common way is to determine maximum likelihood estimates. This method consists in maximizing a function (the likelihood function), which, in the case of the model $(\star)$, writes :

$$ \ell \big( \mu,\sigma \big) = \frac{1}{\big( \sigma \sqrt{2\pi} \big)^{n}} \exp \Big( - \frac{1}{2\sigma^{2}} \sum_{i=1}^{n} (x_{i} - \mu)^{2} \Big) $$

Instead of maximizing $\ell$, it is more convenient to maximize $\log \ell$, the log-likelihood. Maximizing $\log \ell$ with respect to $\mu$ (consider $\sigma$ fixed) yields :

$$ \begin{align*} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \log \ell & = {} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i}-\mu \big)^{2} \\ &= \hat{\mu}_{n} \\ \end{align*} $$

So, our least squares estimates is also (in this case) the maximum likelihood estimate of the mean.


Not the answer you're looking for? Browse other questions tagged or ask your own question.