The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the best linear unbiased estimator (BLUE) for the coefficients in a linear regression (given some conditions on the errors). I can understand why we want an unbiased and minimum-variance ("best") estimator, but why linear? Why not an estimator that depends on any other power (square, square root, etc) of the data?

More specifically, for an $n\times m$ data matrix $X$ predicting an $n \times 1$ response vector $y$ in the model $y = \beta X + \epsilon$, the OLS estimator for the coefficients $\beta$ is,

$$\hat\beta = (X^TX)^{-1}X^Ty = Cy = c_0 y_0 + c_1 y_1 + c_2 y_2 + \cdots $$

Note that $\hat\beta$ is defined linearly in terms of $y_i$ and thus a linear estimator. Is there a particular reason we don't consider estimators of the form, $$ \tilde\beta = Cy^a = c_0 y_0^a + c_1 y_1^a + c_2 y_2^a + \cdots $$

  • $\begingroup$ If $y$ is an $n \times 1$ response vector what would $y^a$ mean? $\endgroup$ – Henry Apr 28 '17 at 7:33
  • $\begingroup$ Edited the original question, but $y^a$ means element-wise exponentiation. $\endgroup$ – user126350 Apr 28 '17 at 18:21

If you have a prior probability distribution of $\beta$ then it is reasonable to use the conditional expected value of $\beta$ given the data as an estimator of $\beta,$ and that is indeed sometimes done. That estimator is nonlinear. It is also biased, since it prefers more probable values of $\beta$ to less probable values.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.