Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

This question is an exact duplicate of:

I think this is a really simple question so please bear with me -- I just had my first class in regression and I'm a little confused about nomenclature/labeling.

Does anyone recommend some good weblinks that explain beginning linear regression really well?

There's a question I've been looking at for a while and I'm not sure how to do it (although I'm sure the solution is simple):

Show that the sample estimate $\hat{\mu} = \bar X = \frac{1}{n} \sum X_i$ is a least square estimator of $\mu$ for a variable $X$ given $X_1, \ldots, X_n$.

My first thought was, $$SSE = \sum (\mu - \hat{\mu})^2,$$

but I'm not sure if that's right. I'm confused about what the $\beta$ is (is it $n$?) and I don't know if there are enough parameters to expand it.

Thanks so much for your patience and if this doesn't make sense, I can clarify more. Thanks!

share|improve this question

migrated from stackoverflow.com Feb 1 at 18:55

This question came from our site for professional and enthusiast programmers.

marked as duplicate by Rahul, egreg, Hagen von Eitzen, Stefan Hansen, Lost1 Feb 3 at 18:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer 1

Let's say that you have $n$ independent and identically distributed (real valued) random variables $(X_{1},\ldots,X_{n})$ with distribution $\mathcal{N}(\mu,\sigma^{2})$. Here, $(\mu,\sigma^{2})$ are unknown parameters which we can estimate. Note that we assume that all random varialbes $X_{i}$ have the same mean $\mu$. In other words, we could say that we assume the following model :

$$ \forall i \in \left\{1,\ldots,n \right\}, \; X_{i} = \mu + \varepsilon_{i} \tag{$\star$}$$

where $(\varepsilon_{1},\ldots,\varepsilon_{n})$ are independent and identically distributed random variables with distribution $\mathcal{N}(0,\sigma^{2})$. One way to estimate the mean $\mu$ is to consider its least squares estimate $\hat{\mu}_{n}$, where $\hat{\mu}_{n}$ is defined as follows :

$$ \hat{\mu}_{n} = \mathop{\mathrm{argmin}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2} $$

To determine this estimate, you want to minimize the function $f$ given by :

$$ f(\mu) = \sum_{i=1}^{n} \big( X_{i} - \mu \big)^{2} = \sum_{i=1}^{n} X_{i}^{2} - 2\mu \sum_{i=1}^{n} X_{i} + n\mu^{2}.$$ $f$ is a differentiable function whose derivative is :

$$ f'(\mu) = -2 \sum_{i=1}^{n} X_{i} + 2n \mu $$

So, $f'(\mu) = 0$ if and only if $\displaystyle \mu = \frac{1}{n} \sum_{i=1}^{n} X_{i}$. Since $\displaystyle \lim \limits_{\mu \to \pm \infty} f(\mu) = +\infty$, the critical point of $f'$ we found is actually the global minimum of $f$. As a conclusion :

$$ \boxed{\displaystyle \hat{\mu}_{n} = \frac{1}{n} \sum_{i=1}^{n} X_{i}} $$


Minimizing a sum of squared "errors" is not the only way to determine the parameters in $(\star)$. The, perhaps, most common way is to determine maximum likelihood estimates. This method consists in maximizing a function (the likelihood function), which, in the case of the model $(\star)$, writes :

$$ \ell \big( \mu,\sigma \big) = \frac{1}{\big( \sigma \sqrt{2\pi} \big)^{n}} \exp \Big( - \frac{1}{2\sigma^{2}} \sum_{i=1}^{n} (x_{i} - \mu)^{2} \Big) $$

Instead of maximizing $\ell$, it is more convenient to maximize $\log \ell$, the log-likelihood. Maximizing $\log \ell$ with respect to $\mu$ (consider $\sigma$ fixed) yields :

$$ \begin{align*} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \log \ell & = {} \mathop{\mathrm{argmax}} \limits_{\mu \in \mathbb{R}} \sum_{i=1}^{n} \big( X_{i}-\mu \big)^{2} \\ &= \hat{\mu}_{n} \\ \end{align*} $$

So, our least squares estimates is also (in this case) the maximum likelihood estimate of the mean.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.