5
$\begingroup$

I'm looking to understand the starred step in the derivation below (also, if someone could help with the LaTex alignment, I'd appreciate it).

The regression line is $y= b_0 + b_1 x$, where $b_0$ and $b_1$ can be found by:

1) taking the difference between each observed value $y_i$ and the expected point regression line, $b_0 + b_1 x_i$ $$\text{ difference } = y_i - b_0 -b_1 x_i$$

2) summing the square of the differences from 1) to get the sum of squares $$SS = \sum \limits_{i=1}^n (y_i - b_0 -b_1 x_i)^2$$ 3) taking the partial derivative with respect to $b_0$ and $b_1$, then solving for each $$ \begin{align} \text{ solving for } b_0 \\ SS &= \sum(y_i - b_0 -b_1 x_i)^2\\ SS &= \sum (Y_i ^2 - 2Y_i b_0 - 2 Y_i b_1+ 2b_0 b_1X_i + b_1^2X_i^2+b_0^2) &\text{expand the square}\\ \frac{ \partial }{\partial_{b_0} }SS &= \sum (-2Y_i + 2b_1 X_i + 2b_0) &\text{partial derivative wrt} b_0\\ 0 &= \sum 2(-Y_i + b_1 X_i + b_0) &\text{factor out 2 from the sum}\\ 0 &= \sum (-Y_i + b_1 X_i + b_0) &\text{divide both sides by 2}\\ 0 &= \sum -Y_i + \sum b_1 X_i + \sum b_0 &\text{split summation into parts}\\ \sum Y_i &= \sum b_1 X_i + \sum b_0 \\ \sum Y_i &= b_1 \sum X_i + n b_0 \\ \frac{1}{n}(\sum Y_i - b_1 \sum X_i ) &= b_0 \\ \bar Y - b_1 \bar X &= b_0 \text { rewrite sums as averages since } \frac{1}{n} \sum Y_i = \bar Y\\ \end{align} $$

$$ \begin{align} \\ \text{solving for } b_1\\ \frac{ \partial }{\partial_{b_1} }SS &= \sum (-2Y_iX_i + 2b_0 X_i + 2 b_1 X_i^2) &\text{ partial derivative wrt } b_1\\ 0 &= 2\sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= \sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= -\sum Y_iX_i + b_0 \sum X_i + b_1 \sum X_i^2 &\text{ split summation}\\ 0 &= -\sum Y_iX_i + (\bar Y - b_1 \bar X) \sum X_i + b_1 \sum X_i^2 &\text{ substitue } b_0\\ 0 &= -\sum Y_iX_i + (\bar Y \sum X_i - b_1 \bar X \sum X_i) + b_1 \sum X_i^2 &\text{ distribute sum}\\ b_1 \bar X \sum X_i - b_1 \sum X_i^2 &= -\sum Y_iX_i + \bar Y \sum X_i &\text{ collect } b_1 \text{ terms}\\ b_1 (\bar X \sum X_i - \sum X_i^2) &= -\sum Y_iX_i + \bar Y \sum X_i \\ b_1 &= { \bar Y \sum X_i -\sum Y_iX_i \over (\bar X \sum X_i - \sum X_i^2) }\\ b_1 &= { \frac{1}{n} \sum Y_i \sum X_i -\sum Y_iX_i \over (\frac{1}{n} \sum X_i \sum X_i - \sum X_i^2) } \biggr ( \frac{-n}{-n} \biggr )\\ b_1 &= { n \sum Y_iX_i - \sum Y_i \sum X_i \over n \sum X_i^2 -(\sum X_i)^2 } \\ \end{align} $$

$$ \begin{align} b_0 &= \frac{1}{n} \sum y_i - b_1 \frac{1}{n} \sum x_i\\\\\\ b_1 &= {n \sum x_i y_i - \sum x_i \sum y_i \over n \sum x_i^2-(\sum x_i)^2} \end{align} $$ (derivation shown in http://polisci.msu.edu/jacoby/icpsr/regress3/lectures/week2/5.LeastSquares.pdf)

From this point you can use $b_1$ to get the correlation coefficient as follows:

$$ \begin{align} b_1 &= {\frac{1}{n} \sum x_i y_i - (\frac{1}{n}\sum x_i) (\frac{1}{n} \sum y_i ) \over (\frac{1}{n} \sum x_i^2) -(\frac{1}{n}\sum x_i)^2} & \text{ divide top and bottom by } n^2 \\\\ b_1 &= {\frac{1}{n} \sum x_i y_i - (\bar x) (\bar y ) \over (\frac{1}{n} \sum x_i^2) -(\bar x)^2} & \text{ rewrite product of sums as averages } \\\\ b_1 &= {\frac{1}{n} \sum (x_i - \bar x)(y_i - \bar y ) \over \frac{1}{n} \sum (x_i - \bar x)^2} & \color{red} *\text{application of inscrutably arcane magic} \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum (x_i - \bar x)^2} } & \text{cancel } \frac{1}{n}\text{, factor denominator }\\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum ( x_i - \bar x)^2} } \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum(y_i - \bar y)^2}}\biggr) & \text{multiply by 1 } \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum(y_i - \bar y)^2}} \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum ( x_i - \bar x)^2} }\biggr) & \text{re-arrange } \\\\ b_1 &= R \frac{S_x}{S_y} \end {align} $$

$\endgroup$

2 Answers 2

2
$\begingroup$

For the numerator, observe that: $$ \begin{align} \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \bar x \bar y &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \dfrac{n}{n}\bar x \bar y & \text{common denominator}\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y \right) & \text{factor out }1/n\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y - n\bar x \bar y + n\bar x \bar y \right) & \text{add $0$ in a fancy way }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x(n\bar y) - \bar y(n\bar x) + n(\bar x \bar y) \right) & \text{rearrange }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x\sum_{i=1}^ny_i - \bar y \sum_{i=1}^n x_i + \sum_{i=1}^n\bar x \bar y \right) & \text{change back to sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_iy_i - \bar xy_i - \bar y x_i + \bar x \bar y) & \text{combine sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_i- \bar x)(y_i - \bar y) & \text{factor}\\ \end {align} $$

As for the denominator: $$ \begin{align} \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - (\bar x)^2 &= \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - \left(\dfrac{n}{n}(\bar x)^2 \right) & \text{common denominator}\\ &= \dfrac{1}{n}\left(\sum_{i=1}^n x_i^2 - n(\bar x)^2 \right) & \text{factor out $1/n$}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2n(\bar x)^2 + n(\bar x)^2 \right)& \text{add $0$ in a fancy way}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x(n\bar x) + n(\bar x)^2 \right)& \text{rearrange}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x \sum_{i=1}^nx_i + \sum_{i=1}^n(\bar x)^2 \right)& \text{change back to sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n \left( x_i^2 - 2\bar xx_i + (\bar x)^2 \right)& \text{combine sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n ( x_i - \bar x)^2 & \text{factor}\\ \end {align} $$

$\endgroup$
2
  • $\begingroup$ The denominator is a special case of the numerator with $y_i=x_i$ and so $\bar y = \bar x$. I also suspect it may be easier to understand why each step happens if you work from the bottom to the top. $\endgroup$
    – Henry
    Feb 11, 2015 at 16:44
  • $\begingroup$ How do we know when to "add 0 in a fancy" way? Is it even possible to provide some pointers or does it just come from experience? I find this to be one of the biggest difficulties when going through derivations. $\endgroup$
    – Falimond
    Oct 7, 2015 at 2:17
2
$\begingroup$

For an alternative method for the same derivation:

$$ \begin{align} \text{solving for } b_1\\ \frac{ \partial }{\partial_{b_1} }SS &= \sum (-2Y_iX_i + 2b_0 X_i + 2 b_1 X_i^2) &\text{ partial derivative wrt } b_1\\ 0 &= 2\sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= \sum(X_iY_i - X_i\bar{Y} + b_1X_i\bar{X}-b_1X_i^2) &\text{replace }b_0 \\ 0 &= \sum(X_iY_i - X_i\bar{Y}) - b_1\sum(X_i^2-X_i\bar{X}) &\text{separate into two sums} \\ b_1 &= \frac{\sum(X_iY_i - X_i\bar{Y})}{\sum(X_i^2-X_i\bar{X})} \\ \end{align} $$

Noting that: $$ \begin{align} \sum X_i\bar{Y} = \sum X_i\left( \frac{1}{n}\sum Y_i \right) = n\left(\frac{1}{n}\sum X_i\right) \left( \frac{1}{n}\sum Y_i \right) = n\bar{X}\bar{Y} \\ \sum X_i\bar{X} = \sum X_i\left( \frac{1}{n}\sum X_i \right) = n\left(\frac{1}{n}\sum X_i\right) \left( \frac{1}{n}\sum X_i \right) = n\bar{X}^2 \: \end{align} $$

$b_1$ can be solved as:

$$ \begin{align} b_1 &= \frac{\sum(X_iY_i) - n\bar{X}\bar{Y}}{\sum(X_i^2)-n\bar{X}^2} \end{align} $$

Finally, to rewrite in a more intuitive form, noting that:

$$ \sum(\bar{X}^2-X_i\bar{X})=0 \: \text{ and } \: \sum(\bar{X}\bar{Y}-Y_i\bar{X})=0 $$

$b_1$ can be written as the ratio of covariance to predictor variance:

$$ b_1 = \frac{\sum(X_iY_i-X_i\bar{Y}) + \sum(\bar{X}\bar{Y}-Y_i\bar{X})}{\sum(X_i^2-X_i\bar{X}) + \sum(\bar{X}^2-X_i\bar{X})} = \frac{\frac{1}{n}\sum(X_i - \bar{X})(Y_i-\bar{Y})}{\frac{1}{n}\sum(X_i-\bar{X})^2} = \frac{Cov(X,Y)}{Var(X)} $$

Ref (equations 6-9):
http://seismo.berkeley.edu/~kirchner/eps_120/Toolkits/Toolkit_10.pdf

$\endgroup$
1
  • $\begingroup$ I've added the essential details, plus a little more detail to help understand one step better. $\endgroup$
    – JStrahl
    Mar 6, 2015 at 14:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .