2
$\begingroup$

Searched quite a bit in the forum but cant find this. In Wikipedia the derivation of the regression coefficient for simple linear regression is skipped and points to a book that isn't freely available.

Instead they just present the results, but I don't find any somewhat friendly derivation, of the steps that go from $\beta_0$, $\beta_1$ to $r_{xy}$.

How is the regression coefficient derived ?

I.e Starting with the first equality for $\beta$ below how do you get to the last definition of $r_{xy}$ ?


Relevant images linked above

enter image description here

I get the steps from the bare linear formula to the one before $r_{xy}$

And this is the definition:

enter image description here

There is also a Wikipedia page for the derivations of quantities but only involved in the least squares error, and does not include the regression coefficient.

$\endgroup$
7
  • 1
    $\begingroup$ So, are you asking how to derive $\widehat\beta$ or how the correlation coefficient $r_{x,y}$ relies to $\widehat\beta$, and what’s the intuition behind it? Just these two questions are a bit different $\endgroup$
    – o.spectrum
    Jun 24, 2023 at 20:17
  • $\begingroup$ How to get from the definition in the image for alpha and beta to the definition for r_{xy}. I have just noticed this seems the pearson correlation coefficient, but still I don't see how to get there from $\alpha$ and $\beta$ definitions. @o.spectrum $\endgroup$
    – user864963
    Jun 24, 2023 at 20:23
  • 1
    $\begingroup$ First, let's note that in the given formula $\widehat\beta$ is expressed as $s_{x,y}/s^2_x$, where $s_{x,y}$ is the sample covariance btw $x$ and $y$, and $s_x$ is just sample standard deviation for $x$. Now, if we talk about the Pearson's correlation coefficient btw $x$ and $y$, then (by definition) $r_{x,y} = s_{x,y} / (s_x \cdot s_y)$, hence $s_{x,y}/s^2_x = (s_y/s_y) \cdot (s_{x,y}/s_x)\cdot 1/s_x = [s_{x,y}/(s_x \cdot s_y)] \cdot s_y / s_x = r_{x,y} \cdot s_y/s_x$. If you want, I can also help you with showing why $r_{x,y}$ is exactly the last formula you gave, as it's not hard to derive $\endgroup$
    – o.spectrum
    Jun 24, 2023 at 20:51
  • $\begingroup$ Thank you. I am able to follow that argument, but not the derivation included in the post @o.spectrum $\endgroup$
    – user864963
    Jun 24, 2023 at 21:12
  • $\begingroup$ can you please specify, what exactly is not clear to you? Is it how $\widehat\beta$ is formed in general? $\endgroup$
    – o.spectrum
    Jun 24, 2023 at 22:18

2 Answers 2

2
$\begingroup$
  • The formula for $\beta$ (after the minimization) is

\begin{align} \beta &= \frac{\mathrm{Cov}(X, Y)}{\mathrm{Var}(X)} \\ &= \frac{\sum_i (x_i - \bar{x})\, (y_i - \bar{y})}{\sum_i (x_i-\bar{x})^2} \end{align}

  1. The Variance can be rewritten (bottom of the post for details)

$$ \mathrm{Var}(X) = n\,(\overline{x^2} - \bar{x}^2)$$

  1. The Covariance can be rewritten

\begin{align} \mathrm{Cov}(X,Y) = n\,(\overline{x\,y} - \bar{x}\bar{y}) \end{align}

  1. Replacing we have:

\begin{align} \beta = \frac{\overline{x\,y} - \bar{x}\bar{y}}{\overline{x^2} - \bar{x}^2} \end{align}

  1. And because we derived $S_x^2$ formulas:

\begin{align} \beta = \left(\frac{\overline{x\,y} - \bar{x}\bar{y}}{\sqrt{(\overline{x^2} - \bar{x}^2)\,(\overline{y^2} - \bar{y}^2)}}\right)\, \frac{S_y}{S_x} \end{align}

  • Where now the term within parens is $r_{xy}$

Some minor details

Formula for the Variance

\begin{align} \sum_i (x_i-\bar{x})^2 &= (\sum_i x_i^2 - 2\,x_i\,\bar{x}) + n\bar{x}^2 \\ &= n\,(\overline{x^2} - 2 \bar{x}^2) + n\, \bar{x}^2 \\ &= n\,(\overline{x^2} - \bar{x}^2) \\ \end{align}

Formula for the Covariance

\begin{align} \mathrm{Cov}(X,Y) &= \sum_i (x_i - \bar{x})\, (y_i - \bar{y}) \\ &= n\,\overline{x\,y} + n\,\bar{x}\bar{y} - (\sum_i x_i\bar{y} + y_i\bar{x}) \\ &= n\,(\overline{x\,y} - \bar{x}\bar{y}) \end{align}

$\endgroup$
0
$\begingroup$

So, let us try to show how both $\widehat\beta_0$ and $\widehat\beta_1$ are derived. Let us assume the following form for the original relation: $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i $$ Here the true regression model parameters $\beta_{0,1}$ are unobserved, so that our task is to estimate them, building a proper estimated relation between $x_i$ and $y_i$: $$ \widehat{y_i} = \widehat{\beta}_0 + \widehat{\beta}_1x_i \\ y_i = \widehat{\beta}_0 + \widehat{\beta}_1 x_i + r_i $$ where $r_i$ is the $i^{\text{th}}$ residual (the difference between the sample observed $y_i$ and the estimated one, $\widehat{y}_i$) that builds up the estimated $y_i$, so that $\widehat{y}_i=y_i - r_i$.

Recall that the regression we are trying to build here is called the “least squares regression”, so essentially our task is to minimize the sum of squared “errors” aka residuals.

Sum of squared errors (SSE) can thus be defined as follows: $$ SSE = \sum\limits_{i=1}^n \left(y_i - \widehat{y}_i\right)^2 = (\text{expanding the estimated } y_i) = \sum\limits_{i=1}^n \left(y_i - \widehat{\beta}_0 - \widehat{\beta}_1x_i\right)^2 $$

Now as we can control only $\widehat{\beta}_0$ and $\widehat{\beta}_1$, let us try to minimize the derived $SSE$ w.r.t. to each of the controllable parameters. It eventually means that we will have to find such $\widehat{\beta}^*_0$ and $\widehat{\beta}^*_1$ that $\left.\frac{\partial SSE}{\partial \widehat{\beta}_0}\right|_{\widehat{\beta}^*_0}=0$, and $\left.\frac{\partial SSE}{\partial \widehat{\beta}_1}\right|_{\widehat{\beta}^*_1}=0$: $$ \frac{\partial SSE}{\partial \widehat{\beta}_0} = \frac{\partial}{\partial{\widehat{\beta}_0}}\left(\sum\limits_{i=1}^n \left(y_i - \widehat{\beta}_0 - \widehat{\beta}_1x_i\right)^2\right) = \frac{\partial}{\partial{\widehat{\beta}_0}}\left(\sum\limits_{i=1}^n \left(\widehat{\beta}_0^2- 2\widehat{\beta}_0y_i+2\widehat{\beta}_0\widehat{\beta}_1x+\dots\right)\right)=2n\widehat{\beta}_0-2\sum\limits_{i=1}^n y_i +2\widehat{\beta}_1 \sum\limits_{i=1}^n x_i;\\ \left.\frac{\partial SSE}{\partial \widehat{\beta}_0}\right|_{\widehat{\beta}^*_0}=0 \Longleftrightarrow 2n\widehat{\beta}^*_0-2\sum\limits_{i=1}^n y_i +2\widehat{\beta}_1 \sum\limits_{i=1}^n x_i =0; \\ 2n \widehat{\beta}^*_0 = 2\sum\limits_{i=1}^n y_i -2\widehat{\beta}_1 \sum\limits_{i=1}^n x_i \Longleftrightarrow \widehat{\beta}^*_0 = \bar y - \widehat{\beta}_1 \bar x. $$

Same can be done for $\widehat{\beta}_1$, so that to find the optimal $\widehat{\beta}^*_1$ that minimizes the SSE. You can try deriving it yourself, it may take a while, but in the end you will get that: $$ \widehat{\beta}^*_1 =\frac{\sum_{i=1}^n (x_i - \bar x)(y_i - \bar y)}{\sum_{i=1}^n(x_i-\bar x)^2}. $$

Getting from there to $r_{x,y}$ in the given form is just the matter of recalling my comment, and of noting that we can express covariance between two variables in the following way: $$ \mathbb{C}\mathrm{ov}(x,y)=\mathbb{E}[xy]- \mathbb{E}[x]\mathbb{E}[y]. $$

Just note that in case of our regression we compute the sample covariance ($s_{x,y}$), not the population covariance $\left(\mathbb{C}\mathrm{ov}(x,y)\right)$, and so instead of $\mathbb{E}[x]$ and $\mathbb{E}[y]$ we deal with sample means: $\bar x$ and $\bar y$.

Hope that my answer was somehow helpful.

$\endgroup$
1
  • $\begingroup$ Thanks, it did help but the question is different I have actually derived the term. It is detailed below. $\endgroup$
    – user864963
    Jun 25, 2023 at 4:27

You must log in to answer this question.