That's my question, I have looking round online and people post a formula by they don't explain the formula. Could anyone please give me a hand with that ? cheers

  • 3
    $\begingroup$ From a geometric point of view, it is almost obvious. The residual vector $r$ is perpendicular to the linear space spanned by column feature vectors, which by convention, contains the intercept vector $1$. Therefore $1^Tr = 0$, and we are done. $\endgroup$ – Zhanxiong Jan 30 '19 at 3:28

If the OLS regression contains a constant term, i.e. if in the regressor matrix there is a regressor of a series of ones, then the sum of residuals is exactly equal to zero, as a matter of algebra.

For the simple regression,
specify the regression model $$y_i = a +bx_i + u_i\,,\; i=1,...,n$$

Then the OLS estimator $(\hat a, \hat b)$ minimizes the sum of squared residuals, i.e.

$$(\hat a, \hat b) : \sum_{i=1}^n(y_i - \hat a - \hat bx_i)^2 = \min$$

For the OLS estimator to be the argmin of the objective function, it must be the case as a necessary condition, that the first partial derivatives with respect to $a$ and $b$, evaluated at $(\hat a, \hat b)$ equal zero. For our result, we need only consider the partial w.r.t. $a$:

$$\frac {\partial}{\partial a} \sum_{i=1}^n(y_i - a - bx_i)^2 \Big |_{(\hat a, \hat b)} = 0 \Rightarrow -2\sum_{i=1}^n(y_i - \hat a - \hat bx_i) = 0 $$

But $y_i - \hat a - \hat bx_i = \hat u_i$, i.e. is equal to the residual, so we have that

$$\sum_{i=1}^n(y_i - \hat a - \hat bx_i) = \sum_{i=1}^n\hat u_i = 0 $$

The above also implies that if the regression specification does not include a constant term, then the sum of residuals will not, in general, be zero.

For the multiple regression,
let $\mathbf X$ be the $n \times k$ matrix containing the regressors, $\hat {\mathbf u}$ the residual vector and $\mathbf y$ the dependent variable vector. Let $\mathbf M = I_n-\mathbf X(\mathbf X'\mathbf X)^{-1}\mathbf X'$ be the "residual-maker" matrix, called thus because we have

$$\hat {\mathbf u} = \mathbf M\mathbf y$$

It is easily verified that $\mathbf M \mathbf X = \mathbf 0$. Also $\mathbf M$ is idempotent and symmetric.

Now, let $\mathbf i$ be a column vector of ones. Then the sum of residuals is

$$\sum_{i=1}^n \hat u_i = \mathbf i'\hat {\mathbf u} =\mathbf i'\mathbf M\mathbf y = \mathbf i'\mathbf M'\mathbf y = (\mathbf M\mathbf i)'\mathbf y = \mathbf 0' \mathbf y = \mathbf 0$$

So we need the regressor matrix to contain a series of ones, so that we get $\mathbf M\mathbf i = \mathbf 0$.

| cite | improve this answer | |
  • $\begingroup$ Thansk a lot, I didnt see your answer till now, really appreciate it :) $\endgroup$ – Maximilian1988 Sep 19 '13 at 4:13
  • $\begingroup$ @Alecos By "constant term", do you mean $\hat{a}$? $\endgroup$ – Stan Shunpike May 17 '16 at 1:21
  • $\begingroup$ @StanShunpike Just $a$ (the unknown quantity) - $\hat a$ is an estimator/estimate. In practice this means that we include as a regressor a series of ones. $\endgroup$ – Alecos Papadopoulos May 17 '16 at 2:20
  • $\begingroup$ How does the regresssor matrix having a series of ones imply $Mi=0$? Thanks for your answer! $\endgroup$ – manofbear Feb 10 '17 at 1:14
  • 3
    $\begingroup$ @manofbear $\mathbf i$ is a series of ones. Part of the properties of $M$ is that if it premultiplies any column of $X$, we get the zero vector. So if a column of ones is not in the regressor matrix, we don't get $\mathbf M \mathbf i=0$. $\endgroup$ – Alecos Papadopoulos Feb 10 '17 at 1:51

The accepted solution by Alecos Papadopoulos has a mistake at the end. I can't comment so I will have to submit this correction as a solution, sorry.

It's true that a series of ones would do the job. But it's not true that we need it. We do not need the regressor to have a series of ones in order for $Mi = 0$.

Theorem: If $\exists$ a $p$ x $1$ vector $v$ such that: $$Xv = 1_n$$

where $1_n$ is a $n$ x $1$ vector of ones, then $$\sum_{i=1}^ne_i=0$$ Proof: $\sum_{i=1}^ne_i= e^T1_n =e^T X v = (e^T X) v=(X^Te)^T v = (0)^T v = 0 $

Above I am using the fact that $X^Te=0$. Having a series of ones in X (a.k.a. intercept) is just a special case of $v$. If the intercept is in the first column $v$ would look like this $[1,0,0,0,0,0...]$

| cite | improve this answer | |

take the estimated values from the line of best fit and use these y values to subtract from the original y values then add them up. if it is a good line of best fit then it should approach zero, but bad lines of best fit will be much less or more than zero

| cite | improve this answer | |

Sum of residuals doesn't exactly equal $0$. However it is a very reasonable assumption that the expectation of the residuals will be $0$. This is similar to the case of unbiased estimation, where we want the bias to be $0$. Here the residual $y_i-\beta_0-\beta_1x_i$ are sometimes negative sometimes positive, but we hope that their overall sum will be $0$, so that the estimation is good enough.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.