3
$\begingroup$

A past lecture introduced the concept of the sum of the squared differences between the dependent variable's mean and its estimated values, $SSR := \sum \left(\hat Y_i - \overline Y\right)^2$. My lecturer's notes offer the additional following formula for the sum of squares due to regression:

$$SSR = a_1 \sum Y_i + b_1 \sum Y_i X_i - n\overline Y^2$$

where $a_1, b_1$ are the coefficients for the regression line $\hat Y_i = a_1 + b_1 X_i$, and $\overline X, \overline Y$ are the arithmetic means for the values of the data set, $X_i, Y_i$.

My lecturer's notes offer no additional explanations (it's a statistics for economics class), and no matter how much I try to tinker with Gauss' normal equations, I can't figure out how exactly they arrive at this result. Would anyone have a reference I could use, or an explanation of how this equation comes about?

$\endgroup$
5
  • $\begingroup$ What is the definition os SSR? Are you sure it is not RSS ("Residual Sum of Squares")? $\endgroup$
    – William M.
    Apr 26 at 14:33
  • $\begingroup$ @WilliamM. I'm pretty sure Residual Sum of Squares and Sum of Squares Residuals are the same thing. $\endgroup$ Apr 26 at 14:39
  • $\begingroup$ @WilliamM. I have it noted as sum of squares due to regression, I will add the exact formula. $\endgroup$
    – shintuku
    Apr 26 at 14:41
  • $\begingroup$ @econbernardo I think they really are different. I know the RSS as $\| y - \hat y\|^2$ while OP wrote SSR as $\|\hat y - \bar y \mathbf{1}\|^2,$ which are very different. In fact, $R^2 = \dfrac{\mathsf{SSR}}{\mathsf{RSS}} \in [0, 1].$ Anyway, I think I knew SSR as TSS ("Total sum of squares"). $\endgroup$
    – William M.
    Apr 26 at 15:32
  • $\begingroup$ This page clarifies the two different sets of terminology. $\sum_i (y_i - \bar{y})^2$ is SST or TSS; $\sum_i (\hat{y}_i - \bar{y})^2$ is SSR or ESS (I have also seen RegSS as in Fox's linear models textbook); $\sum_i (\hat{y}_i - y_i)^2$ is ESS or RSS. $\endgroup$
    – angryavian
    Apr 26 at 21:47

1 Answer 1

2
+50
$\begingroup$

It is often worth it to write these models as a linear expression in lieu of expanding every possible sum. As is usual, I will write variables in lower case and matrices in upper case, otherwise notation will be a mess.

The model is $E(y) = X\beta,$ where $X$ is a matrix of full rank. We want to find the $\beta$ such that $\|y - X\beta\|^2$ is minimum. Calculus yields $$ \hat \beta = (X^\intercal X)^{-1} X^\intercal y $$ In your case, $X = [\mathbf{1}, x]$ where $x$ is your measurements (which you denoted with $X_i$) and $\mathbf{1}$ is a vector of ones. It is easy to see that $$ (X^\intercal X)^{-1} = \left[\begin{matrix} n &n\bar x\\n \bar x & x^\intercal x\end{matrix}\right]^{-1} = \dfrac{1}{n(x^\intercal x - n \bar x^2)} \left[\begin{matrix} x^\intercal x &-n\bar x \\-n \bar x&n\end{matrix}\right] $$ and $$ X^\intercal y = \left[\begin{matrix} n \bar y \\ x^\intercal y\end{matrix}\right], $$ so then $\hat \beta$ is $$ \hat \beta = \dfrac{1}{x^\intercal x - n \bar x^2}\left[\begin{matrix} (x^\intercal x) \bar y - (x^\intercal y) \bar x \\ x^\intercal y - n \bar x \bar y\end{matrix}\right]. $$ If $\hat \beta = (a, b)^\intercal,$ the previous solution reduces to $$ a = \bar y - \bar x b, \quad b = \dfrac{x^\intercal y - n \bar x \bar y}{x^\intercal x - n \bar x} = \dfrac{s_{xy}}{s_{xx}}, $$ where, for given vectors $v$ and $w,$ we write $$ s_{vw} = \dfrac{v^\intercal w - n \bar v \bar w}{n}, $$ which is the most common estimate of the covariance between them.

Now, by definition, the "Total Sums of Squares" is (what you called SSR and) given by $\mathsf{TSS} = \|\hat y - \bar y\mathbf{1}\|^2.$ In your case, this amounts to $$ \|\hat y - \bar y \mathbf{1}\|^2 = \|(a- \bar y) \mathbf{1} +bx\| = (a- \bar y)^2 \mathbf{1}^\intercal \mathbf{1} + 2(a - \bar y)b \mathbf{1}^\intercal x + b^2 x^\intercal x, $$ this simplifies to $$ (\bar x b)^2 n - 2(\bar x b) b n \bar x + b^2 x^\intercal x = b^2 (x^\intercal x - n \bar x^2) = \dfrac{s_{xy}^2}{s_{xx}^2} n s_{xx} = \dfrac{n s_{xy}^2}{s_{xx}}. $$

Now, your expression is $$ a n \bar y + b x^\intercal y - n \bar y^2 = (\bar y-\bar x b) n \bar y + b x^\intercal y - n \bar y^2 = b(x^\intercal y - n \bar x \bar y) = \dfrac{s_{xy}}{s_{xx}} n s_{xy} = \dfrac{n s_{xy}^2}{s_{xx}}. $$ Both expressions coincide. QED

Ammend. Apparently, the total sum of squares is actually $\|y - \bar y \mathbf{1}\|^2$ and what you call the sums of squares due to regression does not appear in my books (e.g. Mardia, Kent and Bibby "Multivariate Analysis"; Seber and Lee "Linear Regression Analysis"; Seber "Multivariate Observations"; Takeuchi, Yanai and Mukherjee "The Foundations of Multivariate Analysis"; Casella and Berger "Statistical Inference"; etc.)

$\endgroup$
3
  • $\begingroup$ Thank you very much for the detailed answer! And I've noted the terminological issue, I will be using total sum of squares! I will be back to award the bounty when the timer ends. $\endgroup$
    – shintuku
    Apr 26 at 16:27
  • $\begingroup$ @shintuku what you wrote as "Sums of Squares due to Regression" and what is usually known as "Total Sums of Squares" are two things. $\endgroup$
    – William M.
    Apr 26 at 16:53
  • $\begingroup$ Noted, I'll sort out these terminological issues. Thanks again! $\endgroup$
    – shintuku
    Apr 26 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.