# Least squares: can't find why $SSR = a_1 \sum Y_i + b_1 \sum Y_i X_i - n\overline Y^2$. Would anyone have references?

A past lecture introduced the concept of the sum of the squared differences between the dependent variable's mean and its estimated values, $$SSR := \sum \left(\hat Y_i - \overline Y\right)^2$$. My lecturer's notes offer the additional following formula for the sum of squares due to regression:

$$SSR = a_1 \sum Y_i + b_1 \sum Y_i X_i - n\overline Y^2$$

where $$a_1, b_1$$ are the coefficients for the regression line $$\hat Y_i = a_1 + b_1 X_i$$, and $$\overline X, \overline Y$$ are the arithmetic means for the values of the data set, $$X_i, Y_i$$.

My lecturer's notes offer no additional explanations (it's a statistics for economics class), and no matter how much I try to tinker with Gauss' normal equations, I can't figure out how exactly they arrive at this result. Would anyone have a reference I could use, or an explanation of how this equation comes about?

• What is the definition os SSR? Are you sure it is not RSS ("Residual Sum of Squares")? Apr 26, 2022 at 14:33
• @WilliamM. I'm pretty sure Residual Sum of Squares and Sum of Squares Residuals are the same thing. Apr 26, 2022 at 14:39
• @WilliamM. I have it noted as sum of squares due to regression, I will add the exact formula. Apr 26, 2022 at 14:41
• @econbernardo I think they really are different. I know the RSS as $\| y - \hat y\|^2$ while OP wrote SSR as $\|\hat y - \bar y \mathbf{1}\|^2,$ which are very different. In fact, $R^2 = \dfrac{\mathsf{SSR}}{\mathsf{RSS}} \in [0, 1].$ Anyway, I think I knew SSR as TSS ("Total sum of squares"). Apr 26, 2022 at 15:32
• This page clarifies the two different sets of terminology. $\sum_i (y_i - \bar{y})^2$ is SST or TSS; $\sum_i (\hat{y}_i - \bar{y})^2$ is SSR or ESS (I have also seen RegSS as in Fox's linear models textbook); $\sum_i (\hat{y}_i - y_i)^2$ is ESS or RSS. Apr 26, 2022 at 21:47

It is often worth it to write these models as a linear expression in lieu of expanding every possible sum. As is usual, I will write variables in lower case and matrices in upper case, otherwise notation will be a mess.

The model is $$E(y) = X\beta,$$ where $$X$$ is a matrix of full rank. We want to find the $$\beta$$ such that $$\|y - X\beta\|^2$$ is minimum. Calculus yields $$\hat \beta = (X^\intercal X)^{-1} X^\intercal y$$ In your case, $$X = [\mathbf{1}, x]$$ where $$x$$ is your measurements (which you denoted with $$X_i$$) and $$\mathbf{1}$$ is a vector of ones. It is easy to see that $$(X^\intercal X)^{-1} = \left[\begin{matrix} n &n\bar x\\n \bar x & x^\intercal x\end{matrix}\right]^{-1} = \dfrac{1}{n(x^\intercal x - n \bar x^2)} \left[\begin{matrix} x^\intercal x &-n\bar x \\-n \bar x&n\end{matrix}\right]$$ and $$X^\intercal y = \left[\begin{matrix} n \bar y \\ x^\intercal y\end{matrix}\right],$$ so then $$\hat \beta$$ is $$\hat \beta = \dfrac{1}{x^\intercal x - n \bar x^2}\left[\begin{matrix} (x^\intercal x) \bar y - (x^\intercal y) \bar x \\ x^\intercal y - n \bar x \bar y\end{matrix}\right].$$ If $$\hat \beta = (a, b)^\intercal,$$ the previous solution reduces to $$a = \bar y - \bar x b, \quad b = \dfrac{x^\intercal y - n \bar x \bar y}{x^\intercal x - n \bar x} = \dfrac{s_{xy}}{s_{xx}},$$ where, for given vectors $$v$$ and $$w,$$ we write $$s_{vw} = \dfrac{v^\intercal w - n \bar v \bar w}{n},$$ which is the most common estimate of the covariance between them.

Now, by definition, the "Total Sums of Squares" is (what you called SSR and) given by $$\mathsf{TSS} = \|\hat y - \bar y\mathbf{1}\|^2.$$ In your case, this amounts to $$\|\hat y - \bar y \mathbf{1}\|^2 = \|(a- \bar y) \mathbf{1} +bx\| = (a- \bar y)^2 \mathbf{1}^\intercal \mathbf{1} + 2(a - \bar y)b \mathbf{1}^\intercal x + b^2 x^\intercal x,$$ this simplifies to $$(\bar x b)^2 n - 2(\bar x b) b n \bar x + b^2 x^\intercal x = b^2 (x^\intercal x - n \bar x^2) = \dfrac{s_{xy}^2}{s_{xx}^2} n s_{xx} = \dfrac{n s_{xy}^2}{s_{xx}}.$$

Now, your expression is $$a n \bar y + b x^\intercal y - n \bar y^2 = (\bar y-\bar x b) n \bar y + b x^\intercal y - n \bar y^2 = b(x^\intercal y - n \bar x \bar y) = \dfrac{s_{xy}}{s_{xx}} n s_{xy} = \dfrac{n s_{xy}^2}{s_{xx}}.$$ Both expressions coincide. QED

Ammend. Apparently, the total sum of squares is actually $$\|y - \bar y \mathbf{1}\|^2$$ and what you call the sums of squares due to regression does not appear in my books (e.g. Mardia, Kent and Bibby "Multivariate Analysis"; Seber and Lee "Linear Regression Analysis"; Seber "Multivariate Observations"; Takeuchi, Yanai and Mukherjee "The Foundations of Multivariate Analysis"; Casella and Berger "Statistical Inference"; etc.)

• Thank you very much for the detailed answer! And I've noted the terminological issue, I will be using total sum of squares! I will be back to award the bounty when the timer ends. Apr 26, 2022 at 16:27
• @shintuku what you wrote as "Sums of Squares due to Regression" and what is usually known as "Total Sums of Squares" are two things. Apr 26, 2022 at 16:53
• Noted, I'll sort out these terminological issues. Thanks again! Apr 26, 2022 at 17:18