30
$\begingroup$

I was asked this question regarding correlation recently, and although it seems intuitive, I still haven't worked out the answer satisfactorily. I hope you can help me out with this seemingly simple question.

Suppose I have three random variables $A$, $B$, $C$. Is it possible to have these three relationships satisfied? $$ \mathrm{corr}[A,B] = 0.9 $$ $$ \mathrm{corr}[B,C] = 0.8 $$ $$ \mathrm{corr}[A,C] = 0.1 $$ My intuition is that it is not possible, although I can't see right now how I can prove this conclusively.

$\endgroup$

5 Answers 5

30
$\begingroup$

Here's an answer to the general question, which I wrote up a while ago. It's a common interview question.

The question goes like this: "Say you have X,Y,Z three random variables such that the correlation of X and Y is something and the correlation of Y and Z is something else, what are the possible correlations for X and Z in terms of the other two correlations?"

We'll give a complete answer to this question, using the Cauchy-Schwarz inequality and the fact that $\mathcal{L}^2$ is a Hilbert space.

The Cauchy-Schwarz inequality says that if x,y are two vectors in an inner product space, then

$$\lvert\langle x,y\rangle\rvert \leq \sqrt{\langle x,x\rangle\langle y,y\rangle}$$

This is used to justify the notion of an ''angle'' in abstract vector spaces, since it gives the constraint

$$-1 \leq \frac{\langle x,y\rangle}{\sqrt{\langle x,x\rangle\langle y,y\rangle}} \leq 1$$ which means we can interpret it as the cosine of the angle between the vectors x and y.

A Hilbert space is an infinite dimensional vector space with an inner product. The important thing for this post is that in a Hilbert space the inner product allows us to do geometry with the vectors, which in this case are random variables. We'll take for granted that the space of mean 0 random variables with variance 1 is a Hilbert space, with inner product $\mathbb{E}[XY]$. Note that, in particular

$$\frac{\langle X,Y\rangle}{\sqrt{\langle X,X\rangle\langle Y,Y\rangle}} = \text{Cor}(X,Y)$$

This often leads people to say that ''correlations are cosines'', which is intuitively true, but not formally correct, as they certainly aren't the cosines we naturally think of (this space is infinite dimensional), but all of the laws hold (like Pythagorean theorem, law of cosines) if we define them to be the negative of the cosines of the angle between two random variables, whose lengths we can think of as their standard deviations in this vector space.

Because this space is a Hilbert space, we can do all of the geometry that we did in high school, such as projecting vectors onto one another, doing orthogonal decomposition, etc. To solve this question, we use orthogonal decomposition, which is often called the ''uncorrelation trick'' in statistics and consists of writing a random variable as a function of another random variable plus a random variable that is uncorrelated with the second random variable. This is especially useful in the case of multivariate normal random variables, when two components being uncorrelated implies independence.

Okay, let's suppose that we know that the correlation of X and Y is $p_{xy}$, the correlation of Y and Z is $p_{yz}$, and we want to know the correlation of X and Z, which we'll call $p_{xz}$. Note that we don't lose generality by assuming mean 0 and variance 1 as scaling and translating vectors doesn't affect their correlations. We can then write that:

$$X = \langle X,Y\rangle Y + O^X_Y$$

$$Z = \langle Z,Y\rangle Y + O^Z_Y$$

where $\langle \cdot,\cdot\rangle$ stands for the inner product on the space and the $O$ are uncorrelated with Y. Then, we take the inner product of $X,Z$ which is the correlation we're looking for, since everything has variance 1. We have that

$$\langle X,Z\rangle = p_{xz} = \langle p_{xy}Y+O^X_Y,p_{zy}Y+O^Z_Y\rangle = p_{xy}p_{yz}+\langle O^X_Y,O^Z_Y\rangle$$

since the variance of Y is 1 and the other terms of this bilinear expansion are orthogonal and hence have 0 covariance. We can now apply the Cauchy-Schwarz inequality to the last term above to get that

$$p_{x,z} \leq p_{xy}p_{yz} + \sqrt{(1-p_{x,y}^2)(1-p_{y,z}^2)}$$

$$p_{x,z} \geq p_{xy}p_{yz} - \sqrt{(1-p_{x,y}^2)(1-p_{y,z}^2)}$$

where the fact that

$$\langle O^X_Y,O^X_Y\rangle = 1-p_{xy}^2$$

comes from the equation setting the variance of X equal to 1 or

$$1 = \langle X,X\rangle = \langle p_{xy}Y + O^X_Y,p_{xy}Y+O^X_Y\rangle = p_{xy}^2 + \langle O^X_Y,O^X_Y\rangle$$

and the exact same thing can be done for $O^Z_Y$.

So we have our answer. Sorry this was so long.

$\endgroup$
5
  • 7
    $\begingroup$ "This often leads people..." This whole paragraph is rather bizarre. First, ''correlations are cosines'' is intuitively true and formally correct. Note that a single correlation involves a 2D vector space and that the three correlations in the question involve a 3D vector space hence no, infinite dimensional spaces are not involved here (and anyway this would not be a problem if they were). Second, no negative of a cosine is involved, only cosines. $\endgroup$
    – Did
    Mar 7, 2017 at 7:08
  • $\begingroup$ "[...] scaling and translating vectors doesn't affect their correlations" If you scale one of the vectors by a negative number it will change the sign of the correlation, though. $\endgroup$
    – Kuku
    Sep 16, 2022 at 14:27
  • 1
    $\begingroup$ @Kuku Here it doesn't mean general scaling and translation. It means standardization by subtracting the mean and dividing by the standard deviation. Since standard deviation is positive, the correlation does not change. $\endgroup$ Nov 29, 2022 at 0:15
  • $\begingroup$ @L.FrancisCong I do not think that "scaling = dividing by the standard deviation" is common usage in the context of vector spaces, and the answer should make it explicit. $\endgroup$
    – Kuku
    Nov 29, 2022 at 10:40
  • $\begingroup$ @Kuku Yeah. I agree. $\endgroup$ Dec 2, 2022 at 14:59
16
$\begingroup$

Assume without loss of generality that the random variables $A$, $B$, $C$ are standard, that is, with mean zero and unit variance. Then, for any $(A,B,C)$ with the prescribed covariances, $$\mathrm{var}(A-B+C)=\mathrm{var}(A)+\mathrm{var}(B)+\mathrm{var}(C)-2\mathrm{cov}(A,B)-2\mathrm{cov}(B,C)+2\mathrm{cov}(A,C), $$ that is, $$ \mathrm{var}(A-B+C)=3-2\cdot0.9-2\cdot0.8+2\cdot0.1=-0.2\lt0, $$ which is absurd.

Edit: Since correlations are cosines, for every random variables such that $\mathrm{corr}(A,B)=b$, $\mathrm{corr}(A,C)=c$ and $\mathrm{corr}(B,C)=a$, one must have $$ a\geqslant bc-\sqrt{1-b^2}\sqrt{1-c^2}. $$ For $b=0.9$ and $c=0.8$, this yields $a\geqslant.458$.

$\endgroup$
10
  • 1
    $\begingroup$ What does "are cosines" mean? $\endgroup$ Jan 23, 2013 at 12:16
  • $\begingroup$ @StefanHansen See Cosines and correlation. $\endgroup$
    – Did
    Jan 23, 2013 at 12:42
  • 3
    $\begingroup$ Say, you accepted the Perfectly possible solution? If ever the problem was asked as a homework and you hand back that, be prepared for some surprise... $\endgroup$
    – Did
    Jan 23, 2013 at 22:49
  • 1
    $\begingroup$ Precisely: geometry tells you that this matrix is not a covariance matrix and that the "complex valued" thing is just wrong. $\endgroup$
    – Did
    Jan 26, 2013 at 2:46
  • 1
    $\begingroup$ @Did cosine similarity is different from pearson correlation when the vectors are not centered. I do not think you can use correlation are cosine here. For example, consider x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18) from wiki. cos(x,y) =0.9208.but cor(x,y)=1. $\endgroup$
    – 81235
    Apr 28, 2019 at 18:10
7
$\begingroup$

As a follow up to one of the other answers, I'd like to finish the solution to make it clear. I find this approach very nice (Borat accent).

In order to generate the correlation matrix, you can leave one of the correlations as $\rho$, I'm leaving $\rho_{AC}$ here. \begin{bmatrix} 1 & 0.9 & \rho\\ 0.9 & 1 & 0.8\\ \rho & 0.8 & 1 \end{bmatrix}

Now, if you calculate the determinant, you'll get the bound for $\rho$. Determinant calculation would result in the following equation (note that the matrix above is a correlation matrix and has to be at least semidefinite):

$$ -0.45 + 0.72\rho + 0.72\rho - \rho^2 \ge 0 $$

making it nicer, it will look like this: $$ \rho^2 - 1.44\rho + 0.45 \le 0 $$

which is correct if $ 0.458 \le \rho \le 0.981 $. This means that the correlation between $A$ and $C$ should be in this range, which isn't.

$\endgroup$
6
$\begingroup$

You can use the fact, that correlations can be understood as cosines between vectors from the common origin. Then apply the arccos-function, and check, whether all possible pairwise sums are greater than the third angle, such that they make a tetraeder. I get

[acos(0.9),acos(0.8),acos(0.1)]
 %1695 = [0.451026811796, 0.643501108793, 1.47062890563]

The sum of the first and the second is smaller than the third, so that combination cannot stem from a trivariate correlation.

$\endgroup$
0
$\begingroup$

For @egreif1 answer, it can be easily understood that $\rho(X, Y) = \rho(X - EX, Y)$, after shifting and scaling ,the correlation between two variables are same. So for the purpose of simplification, we can work with zero mean and unit variance r.v.s. It may be another thought about Hilbert's space.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .