help understanding step in derivation of correlation coefficient

I'm looking to understand the starred step in the derivation below (also, if someone could help with the LaTex alignment, I'd appreciate it).

The regression line is $y= b_0 + b_1 x$, where $b_0$ and $b_1$ can be found by:

1) taking the difference between each observed value $y_i$ and the expected point regression line, $b_0 + b_1 x_i$ $$\text{ difference } = y_i - b_0 -b_1 x_i$$

2) summing the square of the differences from 1) to get the sum of squares $$SS = \sum \limits_{i=1}^n (y_i - b_0 -b_1 x_i)^2$$ 3) taking the partial derivative with respect to $b_0$ and $b_1$, then solving for each \begin{align} \text{ solving for } b_0 \\ SS &= \sum(y_i - b_0 -b_1 x_i)^2\\ SS &= \sum (Y_i ^2 - 2Y_i b_0 - 2 Y_i b_1+ 2b_0 b_1X_i + b_1^2X_i^2+b_0^2) &\text{expand the square}\\ \frac{ \partial }{\partial_{b_0} }SS &= \sum (-2Y_i + 2b_1 X_i + 2b_0) &\text{partial derivative wrt} b_0\\ 0 &= \sum 2(-Y_i + b_1 X_i + b_0) &\text{factor out 2 from the sum}\\ 0 &= \sum (-Y_i + b_1 X_i + b_0) &\text{divide both sides by 2}\\ 0 &= \sum -Y_i + \sum b_1 X_i + \sum b_0 &\text{split summation into parts}\\ \sum Y_i &= \sum b_1 X_i + \sum b_0 \\ \sum Y_i &= b_1 \sum X_i + n b_0 \\ \frac{1}{n}(\sum Y_i - b_1 \sum X_i ) &= b_0 \\ \bar Y - b_1 \bar X &= b_0 \text { rewrite sums as averages since } \frac{1}{n} \sum Y_i = \bar Y\\ \end{align}

\begin{align} \\ \text{solving for } b_1\\ \frac{ \partial }{\partial_{b_1} }SS &= \sum (-2Y_iX_i + 2b_0 X_i + 2 b_1 X_i^2) &\text{ partial derivative wrt } b_1\\ 0 &= 2\sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= \sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= -\sum Y_iX_i + b_0 \sum X_i + b_1 \sum X_i^2 &\text{ split summation}\\ 0 &= -\sum Y_iX_i + (\bar Y - b_1 \bar X) \sum X_i + b_1 \sum X_i^2 &\text{ substitue } b_0\\ 0 &= -\sum Y_iX_i + (\bar Y \sum X_i - b_1 \bar X \sum X_i) + b_1 \sum X_i^2 &\text{ distribute sum}\\ b_1 \bar X \sum X_i - b_1 \sum X_i^2 &= -\sum Y_iX_i + \bar Y \sum X_i &\text{ collect } b_1 \text{ terms}\\ b_1 (\bar X \sum X_i - \sum X_i^2) &= -\sum Y_iX_i + \bar Y \sum X_i \\ b_1 &= { \bar Y \sum X_i -\sum Y_iX_i \over (\bar X \sum X_i - \sum X_i^2) }\\ b_1 &= { \frac{1}{n} \sum Y_i \sum X_i -\sum Y_iX_i \over (\frac{1}{n} \sum X_i \sum X_i - \sum X_i^2) } \biggr ( \frac{-n}{-n} \biggr )\\ b_1 &= { n \sum Y_iX_i - \sum Y_i \sum X_i \over n \sum X_i^2 -(\sum X_i)^2 } \\ \end{align}

\begin{align} b_0 &= \frac{1}{n} \sum y_i - b_1 \frac{1}{n} \sum x_i\\\\\\ b_1 &= {n \sum x_i y_i - \sum x_i \sum y_i \over n \sum x_i^2-(\sum x_i)^2} \end{align} (derivation shown in http://polisci.msu.edu/jacoby/icpsr/regress3/lectures/week2/5.LeastSquares.pdf)

From this point you can use $b_1$ to get the correlation coefficient as follows:

\begin{align} b_1 &= {\frac{1}{n} \sum x_i y_i - (\frac{1}{n}\sum x_i) (\frac{1}{n} \sum y_i ) \over (\frac{1}{n} \sum x_i^2) -(\frac{1}{n}\sum x_i)^2} & \text{ divide top and bottom by } n^2 \\\\ b_1 &= {\frac{1}{n} \sum x_i y_i - (\bar x) (\bar y ) \over (\frac{1}{n} \sum x_i^2) -(\bar x)^2} & \text{ rewrite product of sums as averages } \\\\ b_1 &= {\frac{1}{n} \sum (x_i - \bar x)(y_i - \bar y ) \over \frac{1}{n} \sum (x_i - \bar x)^2} & \color{red} *\text{application of inscrutably arcane magic} \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum (x_i - \bar x)^2} } & \text{cancel } \frac{1}{n}\text{, factor denominator }\\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum ( x_i - \bar x)^2} } \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum(y_i - \bar y)^2}}\biggr) & \text{multiply by 1 } \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum(y_i - \bar y)^2}} \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum ( x_i - \bar x)^2} }\biggr) & \text{re-arrange } \\\\ b_1 &= R \frac{S_x}{S_y} \end {align}

For the numerator, observe that: \begin{align} \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \bar x \bar y &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \dfrac{n}{n}\bar x \bar y & \text{common denominator}\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y \right) & \text{factor out }1/n\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y - n\bar x \bar y + n\bar x \bar y \right) & \text{add 0 in a fancy way }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x(n\bar y) - \bar y(n\bar x) + n(\bar x \bar y) \right) & \text{rearrange }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x\sum_{i=1}^ny_i - \bar y \sum_{i=1}^n x_i + \sum_{i=1}^n\bar x \bar y \right) & \text{change back to sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_iy_i - \bar xy_i - \bar y x_i + \bar x \bar y) & \text{combine sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_i- \bar x)(y_i - \bar y) & \text{factor}\\ \end {align}

As for the denominator: \begin{align} \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - (\bar x)^2 &= \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - \left(\dfrac{n}{n}(\bar x)^2 \right) & \text{common denominator}\\ &= \dfrac{1}{n}\left(\sum_{i=1}^n x_i^2 - n(\bar x)^2 \right) & \text{factor out 1/n}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2n(\bar x)^2 + n(\bar x)^2 \right)& \text{add 0 in a fancy way}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x(n\bar x) + n(\bar x)^2 \right)& \text{rearrange}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x \sum_{i=1}^nx_i + \sum_{i=1}^n(\bar x)^2 \right)& \text{change back to sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n \left( x_i^2 - 2\bar xx_i + (\bar x)^2 \right)& \text{combine sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n ( x_i - \bar x)^2 & \text{factor}\\ \end {align}

• The denominator is a special case of the numerator with $y_i=x_i$ and so $\bar y = \bar x$. I also suspect it may be easier to understand why each step happens if you work from the bottom to the top. – Henry Feb 11 '15 at 16:44
• How do we know when to "add 0 in a fancy" way? Is it even possible to provide some pointers or does it just come from experience? I find this to be one of the biggest difficulties when going through derivations. – Falimond Oct 7 '15 at 2:17

For an alternative method for the same derivation:

\begin{align} \text{solving for } b_1\\ \frac{ \partial }{\partial_{b_1} }SS &= \sum (-2Y_iX_i + 2b_0 X_i + 2 b_1 X_i^2) &\text{ partial derivative wrt } b_1\\ 0 &= 2\sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= \sum(X_iY_i - X_i\bar{Y} + b_1X_i\bar{X}-b_1X_i^2) &\text{replace }b_0 \\ 0 &= \sum(X_iY_i - X_i\bar{Y}) - b_1\sum(X_i^2-X_i\bar{X}) &\text{separate into two sums} \\ b_1 &= \frac{\sum(X_iY_i - X_i\bar{Y})}{\sum(X_i^2-X_i\bar{X})} \\ \end{align}

Noting that: \begin{align} \sum X_i\bar{Y} = \sum X_i\left( \frac{1}{n}\sum Y_i \right) = n\left(\frac{1}{n}\sum X_i\right) \left( \frac{1}{n}\sum Y_i \right) = n\bar{X}\bar{Y} \\ \sum X_i\bar{X} = \sum X_i\left( \frac{1}{n}\sum X_i \right) = n\left(\frac{1}{n}\sum X_i\right) \left( \frac{1}{n}\sum X_i \right) = n\bar{X}^2 \: \end{align}

$b_1$ can be solved as:

\begin{align} b_1 &= \frac{\sum(X_iY_i) - n\bar{X}\bar{Y}}{\sum(X_i^2)-n\bar{X}^2} \end{align}

Finally, to rewrite in a more intuitive form, noting that:

$$\sum(\bar{X}^2-X_i\bar{X})=0 \: \text{ and } \: \sum(\bar{X}\bar{Y}-Y_i\bar{X})=0$$

$b_1$ can be written as the ratio of covariance to predictor variance:

$$b_1 = \frac{\sum(X_iY_i-X_i\bar{Y}) + \sum(\bar{X}\bar{Y}-Y_i\bar{X})}{\sum(X_i^2-X_i\bar{X}) + \sum(\bar{X}^2-X_i\bar{X})} = \frac{\frac{1}{n}\sum(X_i - \bar{X})(Y_i-\bar{Y})}{\frac{1}{n}\sum(X_i-\bar{X})^2} = \frac{Cov(X,Y)}{Var(X)}$$

Ref (equations 6-9):
http://seismo.berkeley.edu/~kirchner/eps_120/Toolkits/Toolkit_10.pdf

• I've added the essential details, plus a little more detail to help understand one step better. – JStrahl Mar 6 '15 at 14:22