The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the best linear unbiased estimator (BLUE) for the coefficients in a linear regression (given some conditions on the errors). I can understand why we want an unbiased and minimum-variance ("best") estimator, but why linear? Why not an estimator that depends on any other power (square, square root, etc) of the data?

More specifically, for an $n\times m$ data matrix $X$ predicting an $n \times 1$ response vector $y$ in the model $y = \beta X + \epsilon$, the OLS estimator for the coefficients $\beta$ is,

$$\hat\beta = (X^TX)^{-1}X^Ty = Cy = c_0 y_0 + c_1 y_1 + c_2 y_2 + \cdots $$

Note that $\hat\beta$ is defined linearly in terms of $y_i$ and thus a linear estimator. Is there a particular reason we don't consider estimators of the form, $$ \tilde\beta = Cy^a = c_0 y_0^a + c_1 y_1^a + c_2 y_2^a + \cdots $$

  • $\begingroup$ If $y$ is an $n \times 1$ response vector what would $y^a$ mean? $\endgroup$ – Henry Apr 28 '17 at 7:33
  • $\begingroup$ Edited the original question, but $y^a$ means element-wise exponentiation. $\endgroup$ – user126350 Apr 28 '17 at 18:21
  • $\begingroup$ G-M is a swindle! Other estimators, e.g. likelihood-based, can be much better under non-normality, and these estimators are not linear functions of the data. G-M only is useful when comparing linear estimators, as in (i) OLS beats WLS under homoscedasticity, (ii) WLS beats OLS under heteroscedasticity (with reasonable weights), and (iii) it is better to use the full data set with OLS or WLS rather than a subset of the full data set with OLS or WLS. G-M does not give much more, even though it is often advertised as stating "OLS is best." $\endgroup$ – BigBendRegion Feb 17 at 19:26

If you have a prior probability distribution of $\beta$ then it is reasonable to use the conditional expected value of $\beta$ given the data as an estimator of $\beta,$ and that is indeed sometimes done. That estimator is nonlinear. It is also biased, since it prefers more probable values of $\beta$ to less probable values.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.