Prove $SST=SSE+SSR$

Prove $$SST=SSE+SSR$$

I start with $$SST= \Sigma (y_i-\bar{y})^2=...=SSE+SSR+ \Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )$$ and I don't know how to prove that $\Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )=0$

a note on notation: the residuals $e_i$ is $e_i=y_i-y_i^*$. A more common notation is $\hat{y}$.

• What is $y_i^*$? And I assume that $\bar{y}$ is the average of the observations $y_1,\ldots,y_n$, but please write such things explicitly in your post. Commented Mar 12, 2014 at 12:16
• $y^*$ is my notation of the often used $\hat{y}$ Commented Mar 12, 2014 at 12:26
• In a nutshell, you have to use the fact that $\sum{e_i} = 0$ and $\sum{\hat y_i e_i} = 0$ (see lectures 3 and 6 at robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures) Commented Mar 15, 2014 at 19:14
• @BradS. Can't see on what slide# Commented Mar 16, 2014 at 12:57
• @jacob Sorry, I should have been more specific. In lecture 3 (robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures/lecture-3/…), he derives the equations for the parameter estimates in simple linear regression and then in lecture 6 (robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures/lecture-6/…) he directly addresses SST = SSR + SSE . In essence, the two fact I mentioned in my previous comment fall out of the minimization he does on the very first (non-title) page of lecture 3. Commented Mar 16, 2014 at 16:22

The principle underlying least squares regression is that the sum of the squares of the errors is minimized. We can use calculus to find equations for the parameters $$\beta_0$$ and $$\beta_1$$ that minimize the sum of the squared errors.

Let $$S = \displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum \left(y_i - \hat{y_i} \right)^2= \sum \left(y_i - \beta_0 - \beta_1x_i\right)^2$$

We want to find $$\beta_0$$ and $$\beta_1$$ that minimize the sum, $$S$$. We start by taking the partial derivative of $$S$$ with respect to $$\beta_0$$ and setting it to zero.

$$\frac{\partial{S}}{\partial{\beta_0}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$

notice that this says, \begin{align}\sum \left(y_i - \beta_0 - \beta_1x_i\right) &= 0 \\ \sum \left(y_i - \hat{y_i} \right) &= 0 \qquad (eqn. 1)\end{align}

Hence, the sum of the residuals is zero (as expected). Rearranging and solving for $$\beta_0$$ we arrive at, \begin{aligned}\sum \beta_0 &= \sum y_i -\beta_1 \sum x_i\\n\beta_0 &= \sum y_i -\beta_1 \sum x_i\\ \beta_0 &= \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i \end{aligned}

now taking the partial of $$S$$ with respect to $$\beta_1$$ and setting it to zero we have, $$\frac{\partial{S}}{\partial{\beta_1}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0$$

and dividing through by $$-2$$ and rearranging we have,

$$\sum x_i \left(y_i - \beta_0 - \beta_1x_i\right) = 0$$ $$\sum x_i \left(y_i - \hat{y_i} \right) = 0$$ but, again we know that $$\hat{y_i} = \beta_0 + \beta_1x_i$$. Thus, $$x_i = \frac{1}{\beta_1}\left( \hat{y_i} - \beta_0 \right) = \frac1{\beta_1}\hat{y_i} -\frac{\beta_0}{\beta_1}$$. Substituting this into the equation above gives the desired result.

\begin{aligned}\sum x_i \left(y_i - \hat{y_i} \right) &= 0\\\sum \left(\frac1{\beta_1}\hat{y_i} - \frac{\beta_0}{\beta_1}\right) \left(y_i - \hat{y_i} \right) &= 0\\\frac1{\beta_1}\sum \hat{y_i} \left(y_i - \hat{y_i} \right) - \frac{\beta_0}{\beta_1} \sum \left(y_i - \hat{y_i} \right)&= 0\end{aligned}

Now, the second term is zero (by eqn. 1) and so, we arrive immediately at the desired result: $$\sum \hat{y_i} \left(y_i - \hat{y_i} \right) = 0 \qquad (eqn. 2)$$

Now, let's use eqn. 1 and eqn. 2 to show that $$\sum \left(\hat{y_i} - \bar{y} \right) \left( y_i - \hat{y_i} \right) = 0$$ - which was your original question.

$$\sum \left(\hat{y_i} - \bar{y} \right) \left( y_i - \hat{y_i} \right) = \sum \hat{y_i} \left( y_i - \hat{y_i} \right) - \bar{y} \sum \left( y_i - \hat{y_i} \right) = 0$$

• Thank you for a detailed answer! A small error: you say "with respect to $\beta_1$ and setting it to zero we have..." but you write $\beta_0$ in the partial. I stopped reading there since your answer was very long and the risk for small errors is big. Is the rest correct? Commented Mar 18, 2014 at 14:38
• @jacob - good catch. I've corrected the typo. The rest is/was correct. Commented Mar 18, 2014 at 16:07
• Perfect answer! Commented Mar 18, 2014 at 16:18
• In the very end of your, did you mean to write $$\sum\left(\hat y_i-\color{red}{\bar y}\right)\left(y_i-\hat y_i\right)=0$$ instead of $$\sum\left(\hat y_i-\color{red}{\bar y_i}\right)\left(y_i-\hat y_i\right)=0\text{ ?}$$ You can't extract $y_i$ with an index in front of the sum over $i'$s. Commented Mar 7 at 15:27
• @PinkyWay, ah, yes. Of course. Thank you for noticing. I've made the necessary edit. Thanks! Commented Mar 8 at 19:37

When an intercept is included in linear regression(sum of residuals is zero), $$SST=SSE+SSR$$.

prove $$\begin{eqnarray*} SST&=&\sum_{i=1}^n (y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i+\hat y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i)^2+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)+\sum_{i=1}^n(\hat y_i-\bar y)^2\\&=&SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y) \end{eqnarray*}$$ Just need to prove last part is equal to 0: $$\begin{eqnarray*} \sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)&=&\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)(\beta_0+\beta_1x_i-\bar y)\\&=&(\beta_0-\bar y)\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)+\beta_1\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)x_i \end{eqnarray*}$$ In Least squares regression, the sum of the squares of the errors is minimized. $$SSE=\displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum_{i=1}^n\left(y_i - \hat{y_i} \right)^2= \sum_{i=1}^n\left(y_i -\beta_0- \beta_1x_i\right)^2$$ Take the partial derivative of SSE with respect to $$\beta_0$$ and setting it to zero. $$\frac{\partial{SSE}}{\partial{\beta_0}} = \sum_{i=1}^n 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$ So $$\sum_{i=1}^n \left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$ Take the partial derivative of SSE with respect to $$\beta_1$$ and setting it to zero. $$\frac{\partial{SSE}}{\partial{\beta_1}} = \sum_{i=1}^n 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0$$ So $$\sum_{i=1}^n \left(y_i - \beta_0 - \beta_1x_i\right)^1 x_i = 0$$ Hence, $$\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=(\beta_0-\bar y)\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)+\beta_1\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)x_i=0$$ $$SST=SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=SSE+SSR$$

similar question: https://stats.stackexchange.com/a/401299/243636

If you have already found the formulas for $$b_0$$ and $$b_1$$, but you are having trouble proving that $$\sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) = 0$$, I think that the following proof is an interesting one:

\begin{aligned} \sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) &= \sum_{i=1}^{n}(y_i - \bar{y} -b_1 (x_i - \bar{x}))(\bar{y} + b_1 (x_i - \bar{x})-\bar{y}) \\ &= b_1 \sum_{i=1}^{n} (y_i - \bar{y})(x_i - \bar{x}) - b_1^2\sum_{i=1}^{n}(x_i - \bar{x})^2 \\ &= b_1 \frac{\sum_{i=1}^{n}(y_i -\bar{y})(x_i - \bar{x})}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \sum_{i=1}^{n}(x_i - \bar{x})^2 - b_1^2\sum_{i=1}^{n}(x_i - \bar{x})^2 \\ &= b_1^2 \sum_{i=1}^{n}(x_i - \bar{x})^2 - b_1^2 \sum_{i=1}^{n}(x_i - \bar{x})^2 \\ &= 0 \end{aligned}

$$2\sum(y_i-y_i^*)(y_i^*-\bar{y})$$ $$= 2\sum[y_i(y_i^*-\bar{y})-y_i^*(y_i^*-\bar{y})]$$ $$= 2\sum Ye_i - 2\bar{Y}\sum e_i$$ $$= 0$$

• I realised I do not get why $y_i$ turns into a random variable $Y$. Also, how could $y_i^*$ turn into the very same random variable $Y$? Commented Mar 14, 2014 at 19:35