10
$\begingroup$

Prove $$SST=SSE+SSR$$

I start with $$SST= \Sigma (y_i-\bar{y})^2=...=SSE+SSR+ \Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )$$ and I don't know how to prove that $\Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )=0$


a note on notation: the residuals $e_i$ is $e_i=y_i-y_i^*$. A more common notation is $\hat{y}$.

$\endgroup$
  • $\begingroup$ What is $y_i^*$? And I assume that $\bar{y}$ is the average of the observations $y_1,\ldots,y_n$, but please write such things explicitly in your post. $\endgroup$ – Stefan Hansen Mar 12 '14 at 12:16
  • $\begingroup$ $y^*$ is my notation of the often used $\hat{y}$ $\endgroup$ – jacob Mar 12 '14 at 12:26
  • 1
    $\begingroup$ In a nutshell, you have to use the fact that $\sum{e_i} = 0$ and $\sum{\hat y_i e_i} = 0$ (see lectures 3 and 6 at robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures) $\endgroup$ – Brad S. Mar 15 '14 at 19:14
  • $\begingroup$ @BradS. Can't see on what slide# $\endgroup$ – jacob Mar 16 '14 at 12:57
  • $\begingroup$ @jacob Sorry, I should have been more specific. In lecture 3 (robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures/lecture-3/…), he derives the equations for the parameter estimates in simple linear regression and then in lecture 6 (robots.ox.ac.uk/~fwood/w4315_fall2010/Lectures/lecture-6/…) he directly addresses SST = SSR + SSE . In essence, the two fact I mentioned in my previous comment fall out of the minimization he does on the very first (non-title) page of lecture 3. $\endgroup$ – Brad S. Mar 16 '14 at 16:22
9
$\begingroup$

The principle underlying least squares regression is that the sum of the squares of the errors is minimized. We can use calculus to find equations for the parameters $\beta_0$ and $\beta_1$ that minimize the sum of the squared errors.

Let $S = \displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum \left(y_i - \hat{y_i} \right)^2= \sum \left(y_i - \beta_0 - \beta_1x_i\right)^2$

We want to find $\beta_0$ and $\beta_1$ that minimize the sum, $S$. We start by taking the partial derivative of $S$ with respect to $\beta_0$ and setting it to zero.

$$\frac{\partial{S}}{\partial{\beta_0}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$

notice that this says, $$\begin{align}\sum \left(y_i - \beta_0 - \beta_1x_i\right) &= 0 \\ \sum \left(y_i - \hat{y_i} \right) &= 0 \qquad (eqn. 1)\end{align}$$

Hence, the sum of the residuals is zero (as expected). Rearranging and solving for $\beta_0$ we arrive at, $$\sum \beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$n\beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$\beta_0 = \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i $$

now taking the partial of $S$ with respect to $\beta_1$ and setting it to zero we have, $$\frac{\partial{S}}{\partial{\beta_1}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0$$

and dividing through by -2 and rearranging we have,

$$\sum x_i \left(y_i - \beta_0 - \beta_1x_i\right) = 0$$ $$\sum x_i \left(y_i - \hat{y_i} \right) = 0$$ but, again we know that $\hat{y_i} = \beta_0 + \beta_1x_i$. Thus, $x_i = \frac{1}{\beta_1}\left( \hat{y_i} - \beta_0 \right) = \frac{1}{\beta_1}\hat{y_i} -\frac{\beta_0}{\beta_1}$. Substituting this into the equation above gives the desired result.

$$\sum x_i \left(y_i - \hat{y_i} \right) = 0 $$ $$\sum \left(\frac{1}{\beta_1}\hat{y_i} - \frac{\beta_0}{\beta_1}\right) \left(y_i - \hat{y_i} \right) = 0$$ $$\frac{1}{\beta_1}\sum \hat{y_i} \left(y_i - \hat{y_i} \right) - \frac{\beta_0}{\beta_1} \sum \left(y_i - \hat{y_i} \right)= 0$$

Now, the second term is zero (by eqn. 1) and so, we arrive immediately at the desired result: $$\sum \hat{y_i} \left(y_i - \hat{y_i} \right) = 0 \qquad (eqn. 2)$$

Now, let's use eqn. 1 and eqn. 2 to show that $\sum \left(\hat{y_i} - \bar{y_i} \right) \left( y_i - \hat{y_i} \right) = 0$ - which was your original question.

$$\sum \left(\hat{y_i} - \bar{y_i} \right) \left( y_i - \hat{y_i} \right) = \sum \hat{y_i} \left( y_i - \hat{y_i} \right) - \bar{y_i} \sum \left( y_i - \hat{y_i} \right) = 0$$

$\endgroup$
  • $\begingroup$ Thank you for a detailed answer! A small error: you say "with respect to $\beta_1$ and setting it to zero we have..." but you write $\beta_0$ in the partial. I stopped reading there since your answer was very long and the risk for small errors is big. Is the rest correct? $\endgroup$ – jacob Mar 18 '14 at 14:38
  • $\begingroup$ @jacob - good catch. I've corrected the typo. The rest is/was correct. $\endgroup$ – Brad S. Mar 18 '14 at 16:07
  • $\begingroup$ Perfect answer! $\endgroup$ – jacob Mar 18 '14 at 16:18
1
$\begingroup$

When an intercept is included in linear regression(sum of residuals is zero), $SST=SSE+SSR$.

prove $$ \begin{eqnarray*} SST&=&\sum_{i=1}^n (y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i+\hat y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i)^2+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)+\sum_{i=1}^n(\hat y_i-\bar y)^2\\&=&SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y) \end{eqnarray*} $$ Just need to prove last part is equal to 0: $$\begin{eqnarray*} \sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)&=&\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)(\beta_0+\beta_1x_i-\bar y)\\&=&(\beta_0-\bar y)\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)+\beta_1\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)x_i \end{eqnarray*} $$ In Least squares regression, the sum of the squares of the errors is minimized. $$ SSE=\displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum_{i=1}^n\left(y_i - \hat{y_i} \right)^2= \sum_{i=1}^n\left(y_i -\beta_0- \beta_1x_i\right)^2 $$ Take the partial derivative of SSE with respect to $\beta_0$ and setting it to zero. $$ \frac{\partial{SSE}}{\partial{\beta_0}} = \sum_{i=1}^n 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0 $$ So $$ \sum_{i=1}^n \left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0 $$ Take the partial derivative of SSE with respect to $\beta_1$ and setting it to zero. $$ \frac{\partial{SSE}}{\partial{\beta_1}} = \sum_{i=1}^n 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0 $$ So $$ \sum_{i=1}^n \left(y_i - \beta_0 - \beta_1x_i\right)^1 x_i = 0 $$ Hence, $$ \sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=(\beta_0-\bar y)\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)+\beta_1\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)x_i=0 $$ $$SST=SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=SSE+SSR$$

similar question: https://stats.stackexchange.com/a/401299/243636

$\endgroup$
-1
$\begingroup$

$$2\sum(y_i-y_i^*)(y_i^*-\bar{y})$$ $$= 2\sum[y_i(y_i^*-\bar{y})-y_i^*(y_i^*-\bar{y})]$$ $$= 2\sum Ye_i - 2\bar{Y}\sum e_i$$ $$= 0$$

$\endgroup$
  • $\begingroup$ I realised I do not get why $y_i$ turns into a random variable $Y$. Also, how could $y_i^*$ turn into the very same random variable $Y$? $\endgroup$ – jacob Mar 14 '14 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.