Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm looking to understand the starred step in the derivation below (also, if someone could help with the LaTex alignment, I'd appreciate it).

The regression line is $y= b_0 + b_1 x$, where $b_0$ and $b_1$ can be found by:

1) taking the difference between each observed value $y_i$ and the expected point regression line, $b_0 + b_1 x_i$ $$\text{ difference } = y_i - b_0 -b_1 x_i$$

2) summing the square of the differences from 1) to get the sum of squares $$SS = \sum \limits_{i=1}^n (y_i - b_0 -b_1 x_i)^2$$ 3) taking the partial derivative with respect to $b_0$ and $b_1$, then solving for each $$ \begin{align} \text{ solving for } b_0 \\ SS &= \sum(y_i - b_0 -b_1 x_i)^2\\ SS &= \sum (Y_i ^2 - 2Y_i b_0 - 2 Y_i b_1+ 2b_0 b_1X_i + b_1^2X_i^2+b_0^2) &\text{expand the square}\\ \frac{ \partial }{\partial_{b_0} }SS &= \sum (-2Y_i + 2b_1 X_i + 2b_0) &\text{partial derivative wrt} b_0\\ 0 &= \sum 2(-Y_i + b_1 X_i + b_0) &\text{factor out 2 from the sum}\\ 0 &= \sum (-Y_i + b_1 X_i + b_0) &\text{divide both sides by 2}\\ 0 &= \sum -Y_i + \sum b_1 X_i + \sum b_0 &\text{split summation into parts}\\ \sum Y_i &= \sum b_1 X_i + \sum b_0 \\ \sum Y_i &= b_1 \sum X_i + n b_0 \\ \frac{1}{n}(\sum Y_i - b_1 \sum X_i ) &= b_0 \\ \bar Y - b_1 \bar X &= b_0 \text { rewrite sums as averages since } \frac{1}{n} \sum Y_i = \bar Y\\ \end{align} $$

$$ \begin{align} \\ \text{solving for } b_1\\ \frac{ \partial }{\partial_{b_1} }SS &= \sum (-2Y_iX_i + 2b_0 X_i + 2 b_1 X_i^2) &\text{ partial derivative wrt } b_1\\ 0 &= 2\sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= \sum (-Y_iX_i + b_0 X_i + b_1 X_i^2) \\ 0 &= -\sum Y_iX_i + b_0 \sum X_i + b_1 \sum X_i^2 &\text{ split summation}\\ 0 &= -\sum Y_iX_i + (\bar Y - b_1 \bar X) \sum X_i + b_1 \sum X_i^2 &\text{ substitue } b_0\\ 0 &= -\sum Y_iX_i + (\bar Y \sum X_i - b_1 \bar X \sum X_i) + b_1 \sum X_i^2 &\text{ distribute sum}\\ b_1 \bar X \sum X_i - b_1 \sum X_i^2 &= -\sum Y_iX_i + \bar Y \sum X_i &\text{ collect } b_1 \text{ terms}\\ b_1 (\bar X \sum X_i - \sum X_i^2) &= -\sum Y_iX_i + \bar Y \sum X_i \\ b_1 &= { \bar Y \sum X_i -\sum Y_iX_i \over (\bar X \sum X_i - \sum X_i^2) }\\ b_1 &= { \frac{1}{n} \sum Y_i \sum X_i -\sum Y_iX_i \over (\frac{1}{n} \sum X_i \sum X_i - \sum X_i^2) } \biggr ( \frac{-n}{-n} \biggr )\\ b_1 &= { n \sum Y_iX_i - \sum Y_i \sum X_i \over n \sum X_i^2 -(\sum X_i)^2 } \\ \end{align} $$

$$ \begin{align} b_0 &= \frac{1}{n} \sum y_i - b_1 \frac{1}{n} \sum x_i\\\\\\ b_1 &= {n \sum x_i y_i - \sum x_i \sum y_i \over n \sum x_i^2-(\sum x_i)^2} \end{align} $$ (derivation shown in http://polisci.msu.edu/jacoby/icpsr/regress3/lectures/week2/5.LeastSquares.pdf)

From this point you can use $b_1$ to get the correlation coefficient as follows:

$$ \begin{align} b_1 &= {\frac{1}{n} \sum x_i y_i - (\frac{1}{n}\sum x_i) (\frac{1}{n} \sum y_i ) \over (\frac{1}{n} \sum x_i^2) -(\frac{1}{n}\sum x_i)^2} & \text{ divide top and bottom by } n^2 \\\\ b_1 &= {\frac{1}{n} \sum x_i y_i - (\bar x) (\bar y ) \over (\frac{1}{n} \sum x_i^2) -(\bar x)^2} & \text{ rewrite product of sums as averages } \\\\ b_1 &= {\frac{1}{n} \sum (x_i - \bar x)(y_i - \bar y ) \over \frac{1}{n} \sum (x_i - \bar x)^2} & \color{red} *\text{application of inscrutably arcane magic} \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum (x_i - \bar x)^2} } & \text{cancel } \frac{1}{n}\text{, factor denominator }\\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum ( x_i - \bar x)^2} } \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum(y_i - \bar y)^2}}\biggr) & \text{multiply by 1 } \\\\ b_1 &= { \sum (x_i - \bar x)(y_i - \bar y ) \over \sqrt{\sum (x_i - \bar x)^2} \sqrt{\sum(y_i - \bar y)^2}} \biggr({\sqrt{\sum(y_i - \bar y)^2} \over \sqrt{\sum ( x_i - \bar x)^2} }\biggr) & \text{re-arrange } \\\\ b_1 &= R \frac{S_x}{S_y} \end {align} $$

share|improve this question

1 Answer 1

up vote 1 down vote accepted

For the numerator, observe that: $$ \begin{align} \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \bar x \bar y &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i \right) - \dfrac{n}{n}\bar x \bar y & \text{common denominator}\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y \right) & \text{factor out }1/n\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - n\bar x \bar y - n\bar x \bar y + n\bar x \bar y \right) & \text{add $0$ in a fancy way }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x(n\bar y) - \bar y(n\bar x) + n(\bar x \bar y) \right) & \text{rearrange }\\ &= \frac{1}{n} \left( \sum_{i=1}^n x_iy_i - \bar x\sum_{i=1}^ny_i - \bar y \sum_{i=1}^n x_i + \sum_{i=1}^n\bar x \bar y \right) & \text{change back to sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_iy_i - \bar xy_i - \bar y x_i + \bar x \bar y) & \text{combine sigmas}\\ &= \frac{1}{n} \sum_{i=1}^n (x_i- \bar x)(y_i - \bar y) & \text{factor}\\ \end {align} $$

As for the denominator: $$ \begin{align} \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - (\bar x)^2 &= \left(\frac{1}{n} \sum_{i=1}^n x_i^2 \right) - \left(\dfrac{n}{n}(\bar x)^2 \right) & \text{common denominator}\\ &= \dfrac{1}{n}\left(\sum_{i=1}^n x_i^2 - n(\bar x)^2 \right) & \text{factor out $1/n$}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2n(\bar x)^2 + n(\bar x)^2 \right)& \text{add $0$ in a fancy way}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x(n\bar x) + n(\bar x)^2 \right)& \text{rearrange}\\ &= \dfrac{1}{n} \left( \sum_{i=1}^n x_i^2 - 2\bar x \sum_{i=1}^nx_i + \sum_{i=1}^n(\bar x)^2 \right)& \text{change back to sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n \left( x_i^2 - 2\bar xx_i + (\bar x)^2 \right)& \text{combine sigmas}\\ &= \dfrac{1}{n} \sum_{i=1}^n ( x_i - \bar x)^2 & \text{factor}\\ \end {align} $$

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.