Let $x,y\in \mathbb{R}^n$ be two vectors.

One way to think about a linear regression on $(x,y)$ is that there is a random variable $X$ and two real numbers $\beta_0$ and $\beta_1$ such that $Y = \beta_0 + \beta_1 X+ \epsilon$ where $\epsilon \sim N(0,\sigma^2)$ is a random Gaussian error. We assume that $(x,y)$ are $n$ samples of the random variables $(X,Y)$. We would like to use the sample $(x,y)$ to best estimate the two real numbers $\beta_0,\beta_1$. The best estimate from the sample $(x,y)$ would be $$ \hat{\beta_0} = \overline{y}-\hat{\beta_1} \overline{x},\quad \hat{\beta_1} = \frac{s_{xy}}{s_{xx}} $$ where $\overline{y} = (\sum_{i = 1}^n y_i)/n$, $\overline{x} = \sum_{i = 1}^n x_i)/n$, $s_{xy} = \sum_{i = 1}^n (x_i-\overline{x})(y_i-\overline{x})$ and $s_{xx} = \sum_{i = 1}^n (x_i-\overline{x})^2$.

The other way to think about the problem is to consider a linear map $\Phi:\mathbb{R}^2\to \mathbb{R}^n$ given by $$ \Phi(b_0,b_1) = b_0\mathbf{1}+b_1x, $$ and we would like to find $(b_0,b_1) \in \mathbb{R}^2$ such that $\|\Phi(b_0,b_1)-y\|_2$ is minimized. Assuming $\mathbf{1}$ and $x$ are linearly independent, it follows that $\Phi$ is injective. This assumption means that the input data $x$ are not all repeats of the same value. By standard inner product spaces argument, there is a unique vector $\hat{y} \in \Phi(\mathbb{R}^2)$ such that $\|y-\hat{y}\|_2$ is minimized. It turns out that $\hat{y} = \hat{\beta_0} \mathbf{1} + \hat{\beta_1} x$, with the same $\hat{\beta_0},\hat{\beta_1}$ as above.

My question is the following: In the first perspective, the natural measure of goodness of fit is given by $$ r^2 =\frac{s_{xy}^2}{s_{xx}s_{yy}} = \left(\frac{\|\hat{y}-\overline{y}\cdot \mathbf{1}\|_2}{\|y-\overline{y}\cdot\mathbf{1}\|_2}\right)^2 = 1 - \frac{\|y-\hat{y}\|_2^2}{\|y-\overline{y}\mathbf{1}\|_2^2}, $$ which can be thought of as percentage of explained variance. Here the third equality is due to the fact that $\hat{y}-\overline{y}\cdot \mathbf{1}$ is orthogonal to $y-\hat{y}$.
However, in the second perspective, the natural measure of goodness of fit is given by $$ \rho^2 = \frac{\|y\|^2_2-\|y-\hat{y}\|_2^2} {\|y\|_2^2} = \left(\frac{\|\hat{y}\|_2}{\|y\|_2}\right)^2 = 1 - \frac{\|y-\hat{y}\|_2^2} {\|y\|_2^2}. $$ Here the second equality holds because $\hat{y}$ and $y-\hat{y}$ are orthogonal.
Why do people often use $r^2$ and not $\rho^2$? What is the name for $\rho^2$ in the literature?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.