A simple question.

When people talk about "the least square estimator", what is this estimator?

Is it an unbiased estimator of the slope of the regression line?

In a paper I'm reading, Let's Take the Con Out of Econometrics, the author writes

Randomization implies that the least squares estimator is "unbiased," but that definitely does not mean that for each sample the estimate is correct. (pg 31, last par)

I understand the second half of the sentence, but I don't understand why "randomization implies that the least squares estimator is 'unbiased.'" I do know that an estimator is unbiased if the expectation of the estimator is equal to the true value of the parameter.

Note: I get the impression that cross validated is for more professional statisticians, not for undergraduate level questions, hence I am posting on this site where I have seen more undergraduate level questions. Is that accurate?

  • 1
    $\begingroup$ I think its fine to ask here, but you can also ask on Cross Validated. I've seen a lot of basic questions there and you'll get a lot of people for whom statistics is their thing. $\endgroup$ – roundsquare Jan 14 at 15:52

In the usual situation you have a model of the form $Y=X\beta+\varepsilon$ where $X$ is a $n \times k$ matrix (where $n$ is the number of samples), $\beta$ is a $k \times 1$ column vector of unknown coefficients (where $k$ is the number of coefficients in the model), and $\varepsilon$ is a $n \times 1$ column vector of noises. In this case the ordinary least squares estimator $\hat{\beta}$ is the minimizer of $\| Y-X\hat{\beta} \|^2$. When $X$ is a full rank matrix (which is typical if $k<n$ and the variables in the model are independent), this minimizer is unique, and is given by $(X^T X)^{-1} X^T Y$. (Note that this is a bad algorithm to use numerically, there are better algorithms for solving a least squares problem numerically.)

That quoted statement is a bit confusingly worded; the point is that the least squares estimator is unbiased, i.e. its expectation is the same as the true value of the parameters, but that it is typically not correct in any particular sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.