4
$\begingroup$

Under the assumptions of the classical simple linear regression model, show that the least squares estimator of the slope is an unbiased estimator of the `true' slope in the model.

Anyone have any ideas for the following questions?

$\endgroup$
11
$\begingroup$

The proof is mostly a matter of a little algebra.

The OLS (ordinary least squares) estimator for $\beta_1$ in the model $$y=\beta_0+\beta_1x+u$$ can be shown to have the form $$\hat{\beta_1}=\frac{\sum(x_i-\bar{x})y_i }{\sum x_i^2-n\bar{x}^2 }$$ Since you didn't say what you've tried, I don't know if you understand how to derive this expression from whatever your book defines $\hat{\beta_1}$ to be.

In any case, taking the conditional expectation with respect to the $x_i$, we get $$E\left[\hat{\beta_1}\left|\right.x_i\right]=\frac{1}{\sum x_i^2-n\bar{x}^2}\sum(x_i-\bar{x})E[y_i|x_i]$$ using the linearity of expectation. But by hypothesis, $$E[y_i|x_i]=E[\beta_0+\beta_1x_i+u|x_i]=\beta_0+\beta_1x_i$$ where we have used the linearity of expectation and the model assumption that $E[u|x_i]=0$. Plugging this back into our last formula, we have

$$E\left[\hat{\beta_1}\left|\right.x_i\right]=\frac{1}{\sum x_i^2-n\bar{x}^2}\sum(x_i-\bar{x})E[y_i|x_i]=\frac{\sum(x_i-\bar{x})(\beta_0+\beta_1x_i)}{\sum x_i^2-n\bar{x}^2}$$

A little algebra on the numerator gives $$\sum (x_i\beta_0+x_i^2\beta_1-\bar{x}\beta_0-\bar{x}x_i\beta_1)=\beta_0\underbrace{\sum x_i}_{=n\bar{x}}-\beta_0\underbrace{\sum\bar{x}}_{=n\bar{x}}+\beta_1\sum\left( x_i^2-x_i\bar{x}\right)$$

So the numerator simplifies to $$\beta_1\sum(x_i^2-x_i\bar{x})=\beta_1\left(\sum x_i^2-\bar{x}\underbrace{\sum x_i}_{=n\bar{x}}\right)=\beta_1\underbrace{\left(\sum x_i^2-n\bar{x}^2\right)}_{\text{the denominator of $E[\hat{\beta_1}|x_i]$}}$$

So we conclude $$\boxed{E\left[\hat{\beta_1}\left|\right.x_i\right]=\beta_1}$$

If you treat the independent variable $X$ as a random variable, then from you here you can use the law of iterated expectations to conclude $$\boxed{E\left[\hat{\beta_1}\right]=E\left[E\left[\hat{\beta_1}\left|\right.x_i\right]\right]=E[\beta_1]=\beta_1}$$ which is what unbiasedness means.

Note that the only model assumptions this proof depends on are

  1. the conditional expectation of $y$ with respect to $x$ is $\beta_0+\beta_1x$
  2. the conditional expectation of $u$ with respect to $x$ is zero
  3. the data points $(x_i,y_i)$ can be thought of as an i.i.d. (independent and identically distributed) random sample from the "true" joint distribution of $x$ and $y$ (actually, you don't need independence, only the identically distributed part)

In particular, you don't need the assumption that the error $u$ is normally distributed. That distributional assumption is required only for inference (i.e. for deriving the sampling distributions of the estimators).

$\endgroup$
  • $\begingroup$ How do you go from The OLS (ordinary least squares) estimator for $\beta_1$ in the model $$y=\beta_0+\beta_1x+u$$ can be shown to have the form $$\hat{\beta_1}=\frac{\sum(x_i-\bar{x})y_i }{\sum x_i^2-n\bar{x}^2 }$$? Since you just show $$\hat{\beta_1}=\frac{\sum(x_i-\bar{x})y_i }{\sum x_i^2-n\bar{x}^2 }$$, how can you just assert it? $\endgroup$ – afsdf dfsaf May 9 '14 at 21:24
  • $\begingroup$ You are confusing several things. (1) There is a difference between the population parameter $\beta_1$ and the estimator $\hat{\beta_1}$. (2) I did not "show" that $\hat{\beta_1}$ has the form I claimed it has; I said it "can be shown." The OLS estimator $\hat{\beta_1}$ has several different but equivalent forms. One is $r\frac{s_y}{s_x}$, where $r$ is the sample correlation between $x$ and $y$, and $s_y$ and $s_x$ are the sample standard deviations. A little algebra shows this is equivalent to the formula I used. $\endgroup$ – symplectomorphic May 9 '14 at 23:17
  • $\begingroup$ This is what I got... I still do not get what you have though?$$\hat{\beta_1} = \frac{S_{xy}}{S_{xx}} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})Y_i -\sum_{i=1}^{n} (x_i - \bar{x})\bar{Y}}{\sum_{i=1}^n (x_i - \bar{x})^2}$$ $\endgroup$ – afsdf dfsaf May 9 '14 at 23:42
  • $\begingroup$ $\sum(x_i-\bar{x})\bar{y}=\bar{y}\sum(x_i-\bar{x})=\bar{y}\cdot 0=0$ and $\sum(x_i-\bar{x})^2=\sum(x_i^2-2x_i\bar{x}+\bar{x}^2)=(\sum x_i^2)-2\bar{x}(\sum x_i)+n\bar{x}^2=(\sum x_i^2)-2\bar{x}(n\bar{x})+n\bar{x}^2=(\sum x_i^2)-n\bar{x}^2$. $\endgroup$ – symplectomorphic May 9 '14 at 23:56
  • 1
    $\begingroup$ No... You can't factor $y_i$ out of the sum. $\endgroup$ – symplectomorphic May 10 '14 at 1:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.