2
$\begingroup$

In probability books it is sometimes mentioned that correlation is a measure of linearity of a relationship between random variables.

This claim is supported by the observation that $\left| \rho(X,Y)\right|=1 \iff X=b+aY$.

But let consider a less extreme case: $X,Y,Z$ are random variables such that $\rho(X,Y)=0.5$ and $\rho(Y,Z)=0.8$. Does it mean that the relationship between $X$ and $Y$ is "less" linear than between $Y$ and $Z$? How at all the degree of linearity in a relationship can be defined (especially given the fact, that a relationship between random variables can be non-functional, so one can not always use the second derivative)?

Edit:

I am talking about Pearson correlation here.

$\endgroup$

7 Answers 7

3
$\begingroup$

For ease of exposition, let us consider zero-mean unit-variance jointly continuous random variables $X$ and $Y$ with joint density $f(x,y)$. Then, $$\begin{align} \rho = E[XY] &= \int_{-\infty}^\infty \int_{-\infty}^\infty x\cdot y\cdot f(x,y)\,\mathrm dx\,\mathrm dy\\ &= \int_0^\infty \int_0^\infty xy\big[f(x,y)+f(-x-y)-f(-x,y)-f(x,-y)\big] \,\mathrm dx\,\mathrm dy. \end{align}$$ If $\rho > 0$, then the term $\big[f(x,y)+f(-x-y)-f(-x,y)-f(x,-y)\big]$ cannot be negative for all $(x,y)$ in the first quadrant; indeed, roughly speaking, $$f(x,y)+f(-x-y) > f(-x,y)+f(x,-y)$$ quite often, that is, there is generally more mass in the vicinity of $(x,y)$ and $(-x-y)$ (i.e. in the first and third quadrants) than in the vicinity of $(-x,y)$ and $(x,-y)$, (i.e. in the second and fourth quadrants). For $\rho$ close to $1$, very roughly speaking, the probability mass lies in the vicinity of the line $x=y$.

For $\rho < 0$ and for $\rho$ close to $-1$, similar remarks apply mutatis mutandis.

The simulations in this answer of Michael Hardy illustrate this notion very well. My point is that this sort of behavior (more mass in two opposite quadrants than in the complementary opposite quadrants when $|\rho|$ is close to $1$) is a general feature shared by all (zero-mean unit-variance) random variables. Similar remarks apply when the mean point is not the origin (the quadrants are defined w.r.t. the mean point) and the variances are different (the slopes of the lines near which the mass lies are different)

$\endgroup$
2
$\begingroup$

It depends on what measure of correlation you use.

The Pearson correlation (the most commonly used one) measures the linearity of a relationship between two random variables.

The Spearman rank correlation however, also measures nonlinear (monotonic) relationships. This is defined as the Pearson correlation coefficient between the ranked variables.

$\endgroup$
1
  • $\begingroup$ Good point, introduced an edit in the OP. $\endgroup$
    – zesy
    Mar 30, 2015 at 14:45
1
$\begingroup$

Perhaps rather than saying "less linear" you might say "less well modeled by a line". If you think of a scatter-plot of $(x,y)$ pairs, it might be sort of a disk or filled ellipse of points, or an amorphous blob. If it's an ellipse, then you'd say that a line drawn along the major axis of the ellipse "sort of fits" the data, especially if the ellipse is highly eccentric. This is exactly the situation in which the correlation is large. If the scatter plot is amorphous, like the standard drawing of an amoeba, then you'd say that it's hard to fit any particular line to the shape; in this case, correlation tends to be small.

I don't think that there's anything deeper here than that; the idea is simply that when the correlation is high, a scatter plot of the data tends to look pretty much like a highly eccentric ellipse, which itself looks a lot like a line.

$\endgroup$
5
  • $\begingroup$ Thanks for that answer, I also never saw anything deeper than the idea you presented, but can we even mathematically show that the higher the correlation is, the more eccentric plot we get (so for $\rho=0,5$ the plot is always going to be less eccentric than for $\rho=0.8$), otherwise how can we be sure, that it is true for all bi-variate distributions? $\endgroup$
    – zesy
    Mar 30, 2015 at 12:28
  • 2
    $\begingroup$ This answer certainly falls short of the whole truth. For example, consider $(x_i:i=1,\ldots,8)$ $= (1,2,3,4,5,6,7,1000)$ and $(y_i:i=1,\ldots,8)$ $=(7,6,5,4,3,2,1,1000)$. My software is telling me the correlation is $0.9999355$. But "pretty much like a highly eccentric ellipse" isn't how I would describe this. ${}\qquad{}$ $\endgroup$ Mar 30, 2015 at 12:46
  • 2
    $\begingroup$ Or, if $(x_i:i=1,\ldots,21)=(-10,-9,-8,\ldots,8,9,10)$ and $y_i=x_i^2$ for $i=1,\ldots,21$ then the correlation is exactly $0$, but it's very far from being an "amorphous blob". ${}\qquad{}$ $\endgroup$ Mar 30, 2015 at 12:49
  • $\begingroup$ I think the idea is that in many cases of actual statistical analysis, when you are looking for relationships between various properties of a set of observations, you often just see an amorphous blob with correlation near zero, and occasionally something that looks like a tilted ellipse or even something close to a diagonal line (where correlation is near 1 or -1). But this seems to me to be an oversimplification. $\endgroup$
    – David K
    Mar 30, 2015 at 13:09
  • $\begingroup$ I agree completely with these comments, although if you tried to draw an ellipse around (or near) Michael's first set of data, I think it'd be pretty highly eccentric, mostly aligned along the $x=y$ line. It was not clear to me at what level the OP was confused, and I figured that a broad-strokes description might be a good place to start. $\endgroup$ Mar 30, 2015 at 13:33
1
$\begingroup$

I've added a graphical answer, and I think many books are deficient in this sort of graphics.

Here's another aspect of the meaning of correlation.

For vectors $(x_1,\ldots,x_n),\ (y_1,\ldots,y_n)\in\mathbb R^n$ suppose the coefficients $\hat\alpha,\hat\beta$ are so chosen as to minimize the sum of squares of the residuals $\hat y_i-y_i$ where $\hat y_i$ is the fitted value $\hat y_i=\hat\alpha + \hat\beta x_i$. ("Residuals" should not be confused with "errors".) Let $\bar y = (y_1+\cdots+y_n)/n$. Then \begin{align} \sum_{i=1}^n (y_i-\bar y)^2 & \text{ is the total sum of squares.} \\[6pt] \sum_{i=1}^n (\hat y_i - \bar y)^2 & \text{ is the explained sum of squares.} \\[6pt] \sum_{i=1}^n (y_i - \hat y_i)^2 & \text{ is the unexplained sum of squares.} \end{align} The total sum of squares it he sum of its explained and unexplained components. (Note that this statement relies on the orthogonality between the vector of fitted values $\hat y_i$ and the vector of residuals $y_i-\hat y_i$, which the reader may verify.)

Then the square of the correlation is the proportion of the total sum of squares that is explained.

$\endgroup$
7
  • $\begingroup$ Thanks, I think it is much closer to what I am looking for, I would only recast your answer in terms of the probability theory: say we have r.v $X,Y$ and we try to approximate $Y$ using $X$, so we decide to do it with a linear function $a+bX$, then we select $a,b$ such that $E(Y-a-bX)^2 \to \min_{a,b \in \mathbb R}$ we find optimal $a ,b$ and then $\rho(X,Y)=\frac{E(b+aX-E(Y))^2}{Var(Y)}$ for the optimal $a,b$, right? $\endgroup$
    – zesy
    Mar 30, 2015 at 13:25
  • $\begingroup$ So $\rho$ can be viewed as a measure of how good one variable can be approximated with another if one does his/her best with a linear function. $\endgroup$
    – zesy
    Mar 30, 2015 at 13:29
  • $\begingroup$ Your last comment is how it is usually stated. But notice that in your statement that $\rho(X,Y)=\text{something}$, they cannot be equal when the correlation is negative. Also, I think the expression on the right side is not right. ${}\qquad{}$ $\endgroup$ Mar 30, 2015 at 13:31
  • $\begingroup$ You need $\dfrac{\operatorname{E}((b+aX - Y)^2)}{\operatorname{var}(Y)} =(\rho(X,Y))^2$, not the version where you have $\operatorname{E}(Y)$. And of course you need to square the correlation. ${}\qquad{}$ $\endgroup$ Mar 30, 2015 at 13:34
  • $\begingroup$ I think the right formula would be $\dfrac{\operatorname{E}((b+aX - Y)^2)}{\operatorname{var}(Y)} =1-(\rho(X,Y))^2$ $\endgroup$
    – zesy
    Mar 30, 2015 at 14:19
1
$\begingroup$

correlation $=0.99$:

correlation $=0.99$

correlation $=0.95$:

correlation $=0.95$

correlation $=0.9$:

correlation $=0.9

correlation $=0.7$:

correlation $=0.7$

correlation $=0.5$:

correlation $=0.5

correlation $=0.3$:

correlation $=0.3$

correlation $=0$:

correlation $=0$

correlation $=-0.3$:

correlation $=-0.3$

correlation $=-0.5$:

correlation $=-0.5$

correlation $=-0.7$:

correlation $=-0.7$

correlation $=-0.9$:

correlation $=-0.9$

correlation $=-0.95$:

correlation $=-0.95$

correlation $=-0.99$:

correlation $=-0.99$

Appendix: Here is how I created these plots.

First I took a sample from a bivariate normal distribution with correlation $0$. This of course gave me a scatterplot with non-zero correlation.

Then I replaced each $y$ value with the corresponding residual $y-\hat y$ where $\hat y$ is the fitted value, i.e. the $y$-coordinate of the least-squares line for predicting $y$ based on $x$. This has two effects: $(1)$ It makes the correlation exactly $0$, and $(2)$ it makes the average of the $y$ values exactly $0$.

Then I divided the $y$ values by the standard deviation of the $y$ values. I also subtracted the average $x$ value from all of them, and then divided them all by their standard deviation. At this point I had a scatterplot in which the two averages were $0$, the two standard deviations were $1$, and the correlation was $0$. That is the plot with correlation $0$ that you see above.

Then, to get a plot with correlation $\rho$, I replaced each data point $\begin{bmatrix} x \\ y \end{bmatrix}$ with $$ \frac 1 2 \begin{bmatrix} \sqrt{1+\rho}+\sqrt{1-\rho}, & \sqrt{1+\rho}-\sqrt{1-\rho} \\ \sqrt{1+\rho}-\sqrt{1-\rho}, & \sqrt{1+\rho}+\sqrt{1-\rho} \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}. $$ That $2\times 2$ matrix is the positive-definite symmetric square root of the correlation matrix $\begin{bmatrix} 1, & \rho \\ \rho, & 1 \end{bmatrix}$.

$\endgroup$
2
  • $\begingroup$ Thanks a lot for the effort you take to explain me the concept, but I do not see how all these graphs address the point of the comment to the first answer. To produce the graphs you simulated a specific parametric bi-variate distribution (I would guess standard normal) with various $\rho$s, and you demonstrated that higher $\rho$ are associated with more eccentric plots. But it does not prove that it is always going to be the case for all bi-variate distributions. So it does not prove that $\rho$ can measure how eccentric a plot for an arbitrary distribution is, can it? $\endgroup$
    – zesy
    Mar 30, 2015 at 13:01
  • $\begingroup$ Actually, if I had simulated a sample of $200$ points from a bivariate normal distribution with correlation $0.7$, it would not be the least bit unusual to get a sample of $200$ points in which the correlation is $<0.65$ or $<0.7$, whereas if the correlation in the scatterplot labeled $0.7$ above is in fact exactly $0.7$, and similarly for the others. ${}\qquad{}$ $\endgroup$ Mar 30, 2015 at 13:24
0
$\begingroup$

Pearson correlation indicates how close the scatter-plot of $(x,y)$ points are to a line. It does not tell you if the relationship between them is linear.

Consider the following graph from Sturdy Econometrics:

Suppose you had planned a regression to estimate the Cuban consumption function and, using income and expenditure data from the Cuban national accounts, you find the estimated equation to be

$$ \mathrm{Expenditure} = 1.9 + 0.69 \times \mathrm{Income} $$

with an $R^2$ of 0.71. This seems like a pretty good equation with a high $R^2$ and a coefficient with the right sign. But suppose further that you had the presence of mind to look at a scatter of observations [as shown below]. The message in these data is very different than the one suggested by the estimated regression. Clearly, the chance that any batch-oriented processing could have picked up this message is very remote indeed. … You would therefore be wise to reserve the right to do some interactive processing and to adjust the way you interpret the data if something unanticipated is discovered.

Cuban Consumption Function

$\endgroup$
9
  • $\begingroup$ "It does not tell you if the relationship between them is linear". it actually does, as I pointed in the OP $\left| \rho(X,Y)\right|=1 \iff X=b+aY$ $\endgroup$
    – zesy
    Mar 30, 2015 at 16:02
  • $\begingroup$ Sure. But if you have $r_{xy} = 0.9$ and $r_{wz} = 0.85$ it doesn't automatically follow that there is a "more linear" relationship between between $x$ and $y$ than there is between $w$ and $z$. It just means $(x,y)$ pairs are closer to a line than are $(w,z)$ pairs. See also en.wikipedia.org/wiki/Correlation_and_dependence#/media/… $\endgroup$ Mar 30, 2015 at 16:16
  • $\begingroup$ I would say, the term "more linear relationship" is so undefined, I would not use it at all, but the argument about how plots will look like for different $\rho$ is totally valid as Dilip Sarwate demonstrated. $\endgroup$
    – zesy
    Mar 30, 2015 at 16:26
  • $\begingroup$ No ... whether the variables are linearly related (except in the case of perfect correlation) cannot be determined via the correlation coefficient. Being close to a straight and being linearly related are different things. $\endgroup$ Mar 30, 2015 at 16:33
  • 1
    $\begingroup$ Maybe you'll understand and enjoy Correlations Genuine and Spurious in Pearson and Yule maybe not. $\endgroup$ Mar 30, 2015 at 17:51
0
$\begingroup$

Late, but it is rather for myself. Let $X,Y$ be random variables. We want to check if one variable can be linearly approximated well enough by another variable. So take the loss function $R(a, b)=E[X-(aY+b)]^2$. We want to find $(a, b)$ such that $R(a, b)$ is minimal. Note that $R(a, b)$ is convex, so a local minimum coincides with the global minimum. $$\frac{\partial R(a,b)}{\partial a}=-2EXY+2aEY^2+2bEY=0$$ $$\frac{\partial R(a,b)}{\partial b}=-2EX+2aEY+2b=0$$ Solving these equations, we get $a^*=\frac{EXY-EXEY}{EY^2-(EY)^2}=\frac{Cov(X,Y)}{Var(Y)},\ b^*=EX-\frac{Cov(X,Y)}{Var(Y)}EY$. Then after some lengthy calculations we get that $$R(a^*,b^*)=Var(X)(1-\frac{Cov(X,Y)^2}{Var(X)Var(Y)})=Var(X)(1-r^2)$$ Where $r=\frac{Cov(X,Y)}{\sqrt{Var(X)}\sqrt{Var(Y)}}$ is the correlation coefficient. When $r=\pm1$ (and $r^2\leq1$ by the Cauchy inequality) we have that $0=R(a^*,b^*)=E[X-(a^*Y+b^*)]^2$, so $X=a^*Y+b^*$ almost surely.

Also $r^2>0$ shows some degree of linear dependence as $X$ can be approximated by a linear function of $Y$ with the loss, which is better than the worst possible case $R(a^*,b^*)=Var(X)$.

As $a^*=r\frac{\sqrt{Var(X)}}{\sqrt{Var(Y)}}$, the correlation coefficient shows positive or negative linear dependence.

And from $R(a^*,b^*)=Var(X)(1-r^2)$ it is clear why $r^2$ is "the proportion of the variance in the dependent variable that is predictable from the independent variable(s)". wiki

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .