2
$\begingroup$

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial derivatives (although I might want to do it wituout partials if it's easier).

I've posted my attempt below. I don't know what to from here.

enter image description here

enter image description here

$\endgroup$
  • 1
    $\begingroup$ You don't know how to solve two equations in two unknowns? $\endgroup$ – Stefan Hansen Mar 18 '14 at 14:35
  • $\begingroup$ Yes I do, I'll try harder and update. $\endgroup$ – jacob Mar 18 '14 at 14:40
  • $\begingroup$ actually, no. I get the wrong answer. $\endgroup$ – jacob Mar 18 '14 at 14:45
2
$\begingroup$

The partial derivatives of $$ Q(\alpha,\beta)=\sum_i (y_i-\alpha-\beta x_i)^2 $$ are $$ \frac{\partial Q}{\partial \alpha}(\alpha,\beta)=-2\sum_i(y_i-\alpha-\beta x_i)=-2(n\bar{y}-n\alpha-n\beta\bar{x})\tag{1} $$ and $$ \frac{\partial Q}{\partial \beta}(\alpha,\beta)=-2\sum_ix_i(y_i-\alpha-\beta x_i)=-2(\mathrm{SP}_{xy}-n\alpha\bar{x}-\beta\mathrm{SS}_x)\tag{2} $$ with $\mathrm{SP}_{xy}=\sum x_i y_i$ and $\mathrm{SS}_x=\sum x_i^2$. Putting $(1)$ equal to zero gives us $$ \hat{\alpha}=\bar{y}-\hat{\beta}\bar{x} $$ and plugging this into $(2)$ gives us the equation $$ \begin{align} 0=\mathrm{SP}_{xy}-n(\bar{y}-\hat{\beta}\bar{x})\bar{x}-\hat{\beta}\mathrm{SS}_x&=\mathrm{SP}_{xy}-n\bar{y}\bar{x}+n\hat{\beta}\bar{x}^2-\hat{\beta}\mathrm{SS}_x\\ &=\mathrm{SP}_{xy}-n\bar{y}-\hat{\beta}(\mathrm{SS}_x-n\bar{x}^2) \end{align} $$ and hence $$ \hat{\beta}=\frac{\mathrm{SP}_{xy}-n\bar{y}\bar{x}}{\mathrm{SS}_x-n\bar{x}^2}. $$

$\endgroup$
1
$\begingroup$

The principle underlying least squares regression is that the sum of the squares of the errors is minimized. We can use calculus to find equations for the parameters $\beta_0$ and $\beta_1$ that minimize the sum of the squared errors, $S$.

$$S = \displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum \left(y_i - \hat{y_i} \right)^2= \sum \left(y_i - \beta_0 - \beta_1x_i\right)^2$$

We want to find $\beta_0$ and $\beta_1$ that minimize the sum, $S$. We start by taking the partial derivative of $S$ with respect to $\beta_0$ and setting it to zero.

$$\frac{\partial{S}}{\partial{\beta_0}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$ $$\sum \left(y_i - \beta_0 - \beta_1x_i\right) = 0 $$ $$\sum \beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$n\beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$\beta_0 = \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i \tag{1}$$ $$\beta_0 = \bar y - \beta_1 \bar x \tag{*} $$

now take the partial of $S$ with respect to $\beta_1$ and set it to zero.
$$\frac{\partial{S}}{\partial{\beta_1}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0$$ $$\sum x_i \left(y_i - \beta_0 - \beta_1x_i\right) = 0$$ $$\sum x_iy_i - \beta_0 \sum x_i - \beta_1 \sum x_i^2 = 0 \tag{2}$$ substitute $(1)$ into $(2)$ $$\sum x_iy_i - \left( \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i\right) \sum x_i - \beta_1 \sum x_i^2 = 0 $$ $$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i + \beta_1 \frac{1}{n} \left( \sum x_i \right) ^2 - \beta_1 \sum x_i^2 = 0 $$

$$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i = - \beta_1 \frac{1}{n} \left( \sum x_i \right) ^2 + \beta_1 \sum x_i^2 $$ $$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i = \beta_1 \left(\sum x_i^2 - \frac{1}{n} \left( \sum x_i \right) ^2 \right) $$ $$\beta_1 = \frac{\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i}{\sum x_i^2 - \frac{1}{n} \left( \sum x_i \right) ^2 } = \frac{cov(x,y)}{var(x)}\tag{*}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.