I think this is a really simple question so please bear with me - I just had my first class in regression and I'm a little confused about nomenclature/labeling.

Does anyone recommend some good weblinks that explain beginning linear regression really well?

There's a question I've been looking at for a while and I'm not sure how to do it (although I'm sure the solution is simple):

Show that the sample estimate $\hat{\mu}(X) = \frac{1}{n} \sum X_i$ is a least square estimator of $\mu$ for a variable $X$ given $X_1, \ldots, X_n$.

My first thought was,

$\mathrm{SSE} = \sum (\mu - \hat{\mu})^2$

But I'm not sure if thats right. I'm confused about what the beta is (is it n?) and I don't know if there are enough parameters to expand it.

Thanks so much for your patience and if this doesn't make sense, I can clarify more. Thanks!


Now let $m$ be the mean and $a$ any estimate.

$$ \sum (x_i -a)^2 = \sum \left( (x_i-m + m -a)^2 \right) =\sum \left( (x_i-m)^2 + 2 (x_i-m) (m-a) + (m-a)^2 \right)\\ =\sum (x_i-m)^2 + 2 (m-a) \sum (x_i-m) + \sum (m-a)^2 $$ Now the basic property of the mean is $\sum(x_i-m) =0$. So $$ \sum (x_i -a)^2 =\sum (x_i-m)^2 + N (m-a)^2 $$ where $N$ is the number of data points. Clearly the first summation does not depend on $a$. The second is always non negative and is zero when $m=a$. So the best estimate is $a=m$.

This also shows that the minimum is $\sum (x_i-m)^2 $ which is the variance.


Least squares estimator is an estimator that minimizes the sum of the squares of the deviation from your observation to the estimate. This means you are seeking a $\hat{\mu}$ that solves the following problem

$\min\{\sum{(X_i-\hat{\mu)^2}}\}$. Now if you have somewhat familiarity with calculus the following will make total sense, if not I suggest you read about derivatives and what are known as the First and Second Order Conditions. In a nutshell, they say that in an unconstrained optimization problem, if the objective function is differentiable (you can take the derivative(s)) a solution is a vector s.t. first derivative of the function w.r.t any component of the vector you're optimizing on, evaluated at solution is zero SOC tells you whether what you just found is a maximum or a minimum. In this case the objective function is differentiable. Hence you have that $2\cdot\sum{(X_i-\hat{\mu})=0}$ which is the same as saying that $\sum X_i-N\hat{\mu}=0$ implying that $\hat{\mu}=(1/N)\cdot\sum X_i$

  • $\begingroup$ Sure, that clears up some of the questions I had on least squares estimators! Thanks. One more thing though - why do you write N in the last equation instead of distribute the summation sign? $\endgroup$ – Jessie Bullock Jan 10 '14 at 7:30
  • $\begingroup$ Because $\sum(X_i-\hat{\mu})=X_1-\hat{\mu} + X_2-\hat{\mu}+\cdots + X_N-\hat{\mu}=\sum X_ i - \hat{\mu}-\hat{\mu}-\cdots - \hat{\mu} = \sum X_i - N\hat{\mu}$ $\endgroup$ – mathemagician Jan 10 '14 at 7:33
  • $\begingroup$ I think what you want to minimize is $S = \langle (\mu - \hat\mu)^2 \rangle$, where $\mu$ is the true mean and $\hat\mu$ is your estimator, which is a function of the observations $X_1,\dots,X_n$. So you have to find the function $\hat\mu$ of the data that minimizes $S$. This is a calculus of variations problem. $\endgroup$ – becko Nov 10 '16 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.