21
$\begingroup$

I'm new to linear regression and am trying to teach myself.

In my textbook there's a problem that asks "why is $R^{2}$ in the regression of $Y$ on $X$ equal to the square of the sample correlation between X and Y?"

I've been throwing my head against this for a while and I keep getting stuck because in the correlation coefficient there is a $X$ and $\bar{X}$ term, whilst in the $R^{2}$ term there is no such thing.

Can anyone provide a derivation as to why $R^{2}$ is the correlation coefficient squared?

Thanks!

$\endgroup$
2
  • 3
    $\begingroup$ It might help if you define the terms in your question. What is the equation for $R^2$, in particular? $\endgroup$
    – user856
    May 10, 2012 at 9:27
  • 1
    $\begingroup$ If by $R^2$ you mean the "explained variance", then stats.SE might be a more suitable site for this question. See, for example, this question or this one for some ideas related to this. $\endgroup$ Jan 4, 2013 at 23:46

5 Answers 5

31
$\begingroup$

Suppose that we have $n$ observations $(x_1,y_1),\ldots,(x_n,y_n)$ from a simple linear regression $$ Y_i=\alpha+\beta x_i+\varepsilon_i, $$ where $i=1,\ldots,n$. Let us denote $\hat y_i=\hat\alpha+\hat\beta x_i$ for $i=1,\ldots,n$, where $\hat\alpha$ and $\hat\beta$ are the ordinary least squares estimators of the parameters $\alpha$ and $\beta$. The coefficient of the determination $r^2$ is defined by $$ r^2=\frac{\sum_{i=1}^n(\hat y_i-\bar y)^2}{\sum_{i=1}^n(y_i-\bar y)^2}. $$ Using the facts that $$ \hat\beta=\frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)}{\sum_{i=1}^n(x_i-\bar x)^2} $$ and $\hat\alpha=\bar y-\hat\beta\bar x$, we obtain \begin{align*} \sum_{i=1}^n(\hat y_i-\bar y)^2 &=\sum_{i=1}^n(\hat\alpha+\hat\beta x_i-\bar y)^2\\ &=\sum_{i=1}^n(\bar y-\hat\beta\bar x+\hat\beta x_i-\bar y)^2\\ &=\hat\beta^2\sum_{i=1}^n(x_i-\bar x)^2\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2\sum_{i=1}^n(x_i-\bar x)^2}{[\sum_{i=1}^n(x_i-\bar x)^2]^2}\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2}{\sum_{i=1}^n(x_i-\bar x)^2}. \end{align*} Hence, \begin{align*} r^2 &=\frac{\sum_{i=1}^n(\hat y_i-\bar y)^2}{\sum_{i=1}^n(y_i-\bar y)^2}\\ &=\frac{[\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)]^2}{\sum_{i=1}^n(x_i-\bar x)^2\sum_{i=1}^n(y_i-\bar y)^2}\\ &=\biggl(\frac{\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_{i=1}^n(x_i-\bar x)^2\sum_{i=1}^n(y_i-\bar y)^2}}\biggr)^2. \end{align*} This shows that the coefficient of determination of a simple linear regression is the square of the sample correlation coefficient of $(x_1,y_1),\ldots,(x_n,y_n)$.

$\endgroup$
2
  • 4
    $\begingroup$ Could anyone explain the reason for the downvote?.. $\endgroup$
    – Cm7F7Bb
    Sep 29, 2017 at 7:48
  • 1
    $\begingroup$ very clear explanation. $\endgroup$ Jul 22, 2019 at 12:03
5
$\begingroup$

The complete proof of how to derive the coefficient of determination $R^{2}$ from the Squared Pearson Correlation Coefficient between the observed values $y_{i}$ and the fitted values $\hat{y}_{i}$ can be found under the following link:

http://economictheoryblog.wordpress.com/2014/11/05/proof/

In my eyes it should be pretty easy to understand, just follow the single steps.

$\endgroup$
0
1
$\begingroup$

There are many forms of the computation available online (such as the Wikipedia page on the correlation coefficient http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Pearson.27s_correlation_and_least_squares_regression_analysis ) but note that this is a magical algebraic property of least squares linear regression, not linear regression in general.

$\endgroup$
2
  • $\begingroup$ hmm what I don't understand about that is why the correlation coeff equation doesn't have an X term any more? $\endgroup$
    – Scubadiver
    Apr 10, 2012 at 5:49
  • $\begingroup$ Which equation does not have an X term? $\endgroup$
    – zyx
    Apr 10, 2012 at 6:03
0
$\begingroup$

There are different forms to express R2: Some expressions have (X-Xbar) squared in the numerator, while others express it just with the square of predicted ys. All forms are equivalent.

References: Dougherty; Gujarati; Wooldridge

$\endgroup$
0
$\begingroup$

The following answer assumes you are familiar with linear algebra and matrix factorization. The basic idea is to express both the Pearson correlation and $R^2$ in terms of vector of observations (and intercept).

Suppose you have made $n$ observations $(x_i, y_i): i = 1, 2, \ldots, n$, denote the column vector $(y_1, y_2, \ldots, y_n)'$ and the column vector $(x_1, x_2, \ldots, x_n)'$ by $y$ and $x$ respectively. Moreover, denote the $n$-long column vector of all ones by $e$. The design matrix $X$ thus can be written as $X = \begin{bmatrix} e & x \end{bmatrix}$. Let $X = QR$ be the QR decomposition of $X$, where $Q = \begin{bmatrix} q_1 & q_2\end{bmatrix}$ is column-wise orthogonal, i.e., $q_1'q_1 = q_2'q_2 = 1$ and $q_1'q_2 = 0$.

With these preparations, it is readily seen that the Pearson correlation $r$ is \begin{align} r = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum(x_i - \bar{x})^2\sum(y_i - \bar{y})^2}} = \frac{x'(I - q_1q_1')y}{\sqrt{x'(I - q_1q_1')x \cdot y'(I - q_1q_1')y}}. \tag{1} \end{align} While the coefficient of determination is given by \begin{align} R^2 = \frac{\sum(\hat{y}_i - \bar{y})^2}{\sum(y_i - \bar{y})^2} = \frac{y'(H - q_1q_1')y}{y'(I - q_1q_1')y} = \frac{(y'q_2)^2}{y'(I - q_1q_1')y}, \tag{2} \end{align} where $H = X(X'X)^{-1}X' = QQ' = q_1q_1' + q_2q_2'$ is the so-called hat matrix. It thus follows by $(1)$ and $(2)$ that to show $R^2 = r^2$, it is sufficient to prove \begin{align} (x'(I - q_1q_1')y)^2 = (y'q_2)^2 \times x'(I - q_1q_1')x. \tag{3} \end{align} Since $x$ is in the space spanned by two orthonormal vectors $q_1$ and $q_2$, we have $x = (x'q_1)q_1 + (x'q_2)q_2$, whence \begin{align} & x'(I - q_1q_1')x = \|x\|^2 - (x'q_1)^2 = (x'q_2)^2, \\ & x'(I - q_1q_1')y = ((x'q_1)q_1' + (x'q_2)q_2')(y - q_1q_1'y) = (x'q_2)q_2'y. \end{align} Hence $(3)$ holds. This completes the proof.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .