Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm having a bit of trouble proving $\sum (x_i - \bar{x})\hat{e_i} = 0$. What I know so far is that the total sum of $\hat{e_i}$'s is zero by property of OLS so when you distribute the $\hat{e_i}$ in, one term "cancels out" and you are left with $\sum x_i\hat{e_i}$ which is equivalent to $\sum x_i(y_i-b_1-b_2x_i)$

When I attempt to simplify more, I keep getting stuck. What I'm doing so far is:

$\sum x_iy_i - b_1\sum x_i - b_2 \sum x_i^2$ = $\sum x_iy_i -(\bar{y}-b_2 \bar{x})\sum x_i - b_2 \sum x_i^2$

= $\sum x_iy_i - \bar{y}\sum x_i + b_2\bar{x}\sum x_i - b_2 \sum x_i^2$. And at this point I get stuck. Is there an easier trick? An OLS property I am forgetting? Thanks!

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

From your notation I assume that your true model is:

$$ Y_i=\beta_1+\beta_2 X_i + \epsilon_i \qquad i=1,\ldots,n $$ where $\beta_1$ and $\beta_2$ are the true parameters of the model. Then, if you assume that the (true) error term is iid and has zero conditional expectation you get (i.e., $E[\epsilon_i|X]=0$):

$$ E[Y_i|X_i]=\beta_1 + \beta_2 X_i + E[\epsilon_i|X_i]\\ =\beta_1+\beta_2 X_i $$ which is what you want to estimate using OLS. Let $b_1$ and $b_2$ be the OLS estimates of $\beta_1$ and $\beta_2$ respectively. Then, $$ b_2=\frac{\sum_{i=1}^{n} (Y_i-\bar{Y})(X_i-\bar{X})}{\sum_{i=1}^{n}(X_i-\bar{X})^2}\\ $$ Since, $$ \sum_{i=1}^{n} (Y_i-\bar{Y})(X_i-\bar{X})=\sum_{i=1}^{n} Y_i(X_i-\bar{X})-\bar{Y}\sum_{i=1}^{n}(X_i-\bar{X})\\ =\sum_{i=1}^{n} Y_i(X_i-\bar{X}) $$ and, $$ \sum_{i=1}^{n} (X_i-\bar{X})(X_i-\bar{X})=\sum_{i=1}^{n} X_i(X_i-\bar{X})-\bar{X}\sum_{i=1}^{n}(X_i-\bar{X})\\ =\sum_{i=1}^{n} X_i(X_i-\bar{X}) $$ because $\sum_i(X_i-\bar{X})=0$ you can rewrite $b_2$ as follows: $$ b_2=\frac{\sum_{i=1}^{n} Y_i(X_i-\bar{X})}{\sum_{i=1}^{n}X_i(X_i-\bar{X})}\\ $$ Let $e_i$ be the OLS residual, i.e., $e_i=Y_i-b_1-b_2X_i$, $i=1,\ldots,n$. Then, $$ \sum_i(X_i-\bar{X})e_i=\sum_i(X_i-\bar{X})(Y_i-b_1-b_2X_i)\\ =\sum_i Y_i(X_i-\bar{X})-b_1\sum_i(X_i-\bar{X})-b_2\sum_i X_i(X_i-\bar{X})\\ =\sum_i Y_i(X_i-\bar{X})-b_2\sum_i X_i(X_i-\bar{X})\\ =\sum_i Y_i(X_i-\bar{X})-\left(\frac{\sum_i Y_i(X_i-\bar{X})}{\sum_i X_i(X_i-\bar{X})}\right)\sum_i X_i(X_i-\bar{X}) $$ where again I have used the fact that $\sum_i (X_i-\bar{X})=0$.

share|improve this answer
add comment

Assume you have a model of the following form: $y=X\beta+\epsilon$ where $X$ is a $n$ by $K$ matrix of $K$ explanatory variables and $n$ observations of each variable. $y$ a column vector consisting of the $n$ observations of the dependent variable and $\epsilon$ is a column vector consisting of the error terms. In your case it seems like you have a model where $X=[1,X_1]$ (i.e. one explanatory variable and an intercept). Now, I assume you know how to find the normal equation when solving for the least squares estimate $b$ of $\beta$. The normal equation reads: $$X^te=0$$ where $e=y-Xb$. Now by simply multiplying out the normal equation above your claim follows, since $0=X^te=[\sum{e_i},\sum{x_ie_i}]^t$. Hence $\sum{(x_i-\bar{x})e_i}=\sum{x_ie_i}-\bar{x}\sum{e_i}=0$.

Also note that the result above applies to any multivariate linear regression model as long as you have an intercept term. However without the intercept term (also called a regression through the origin) we can not be sure that $\sum{e_i}=0$ and hence your claim may no longer be true.

share|improve this answer
    
Thanks. I really liked your explanation. I didn't even think to do it in a more "linear algebra" type approach. I'm still trying to get used to visualizing where summations of products fit come from matrices. –  user12345121212 Jul 10 '12 at 18:08
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.