3
$\begingroup$

I am studying from a book that asserts that the correlation between random variables $X$ and $Y$ is 0.8. And apparently this means that if $X$ and $Y$ are represented on a vector space, with angle $\theta$ between them, then $\cos(\theta) = 0.8$.

I don't understand why this is true if $X$ and $Y$ are not centered random variables. For example, what if we consider random variables $A$ and $B$ such that $A$ can be represented by the vector $[0 \ \ 1]^T$ and $B$ by $[1 \ \ 0]^T$. They are clearly orthogonal with $\theta = \frac{\pi}{2}$, so according to the logic in the first paragraph, the correlation should be zero.

But the correlation isn't zero because the covariance isn't zero. The covariance is $$ \text{cov}(A,B) = E[(A - \mu_A)(B - \mu_B)] \\ = \frac{1}{2}\sum_{i=1}^2 (a_i - \mu_A)(b_i - \mu_B) \\ = \frac{1}{2}\left[ -0.5 \cdot 0.5 \ + \ 0.5\cdot-0.5 \right] \\ = -0.25 $$

So based on this simple example, it seems $\rho_{AB} \neq cos(\theta_{AB})$


Here is a screenshot of the part of the book that I was referring to above. I think everything in the screenshot is wrong unless those random variables have ZERO mean.

\ enter image description here

$\endgroup$
  • $\begingroup$ Orthogonality and uncorrelatedness (and also independence) are different concepts. In particular, orthogonality does not imply zero correlation. On the other hand, as you correctly noted, whether orthogonal variables are correlated or non-correlated depends on whether the expectations are zero or not. This directly results from the definition of covariance. $\endgroup$ – Anatoly Jul 24 at 10:46
  • $\begingroup$ @Anatoly Thanks. Yes, I think I agree with that. Orthogonality doesn't imply zero correlation, and I also not believe zero correlation implies orthogonality. If this is true then it seems the approach in the above problem is incorrect? As you can see, it seems to assume that when the correlation coefficient is the cosine of the angle, but that shouldn't be true if the expectation isn't zero? $\endgroup$ – David Jul 24 at 18:02
  • $\begingroup$ You are right. A commonly used term is "cosine similarity", which can be roughly interpreted as the cosine of the angle between the two vectors. Notably, cosine similarity is not invariant to shifts. Correlation can be seen as the cosine similarity between the "centered" versions of the vectors. $\endgroup$ – Anatoly Jul 24 at 19:04
  • $\begingroup$ @Anatoly Ah I see. The solution they obtained under what appears to be a false premise does match the solution (using another approach) exactly. So I'm wondering if there's some theoretical guarantee that assuming $\rho = \cos(\theta)$ gives the exact solution. $\endgroup$ – David Jul 24 at 21:25
  • $\begingroup$ I wrote an answer to better explain the similarities and the differences between cosine similarity and correlation coefficient. $\endgroup$ – Anatoly Jul 24 at 23:39
1
+50
$\begingroup$

As already noted in the comments, the concepts of cosine similarity and correlation are different. In particular, as explained below, the cosine of the angle between two vectors can be considered equivalent to the correlation coefficient only if the random variables have zero means. This explains why two orthogonal vectors, whose cosine similarity is zero, can show some correlation, and then a covariance different from zero as in the example of the OP.

Cosine similarity is obtained by taking the inner product and dividing it by the vectors’ $L2$ norms. The formula is

$${\displaystyle CS(x,y) ={\frac {\sum \limits _{i=1}^{n}{x_{i}x_{i}}}{{\sqrt {\sum \limits _{i=1}^{n}{x_{i}^{2}}}}{\sqrt {\sum \limits _{i=1}^{n}{y_{i}^{2}}}}}}= {\langle x,y \rangle \over \| x \|\|{y} \|} }$$

and corresponds to the cosine of the angle between the two vectors. Cosine similarity is bounded between $-1$ and $1$. However, in most applications where this measure is used, the vectors are non-negative, so in these cases it ranges between $0$ and $1$. Importantly, cosine similarity is invariant to scaling (i.e. multiplying all terms by a nonzero constant) but is not invariant to shifts (i.e. adding a constant to all terms).

On the other hand, correlation can be seen as the cosine similarity measured between the centered versions of the two vectors. In fact, indicating with $\overline{x}$ and $\overline{y}$ the means, we have

$${\displaystyle r(x,y) ={\frac {\sum \limits _{i=1}^{n}({x_{i}-\overline{x})(y_{i}- \overline{y} ) }}{{\sqrt {\sum \limits _{i=1}^{n}{ (x_{i}-\overline{x}) ^{2}}}}{\sqrt {\sum \limits _{i=1}^{n}{ (y_{i}-\overline{y})^{2}}}}}}} = {\langle x-\overline{x}, \,y -\overline{y}\rangle \over \| x-\overline{x} \|\|{y}-\overline{y} \|} $$

and then

$$r(x,y)=CS(x-\overline{x}, \,y -\overline{y})$$

It is worthy of note that correlation is bounded between $-1$ and $1$ as well, but unlike cosine similarity it is invariant to both scaling and shifts.

We conclude that the cosine similarity is equal to the correlation coefficient only when the vectors $x$ and $y$ are centered (i.e., they have zero means).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 Thanks for writing out this detailed explanation. But I'm still confused by the solution in the screenshot of the book in the OP. Is it correct to say that they used a false premise (assuming cosine similarity is the same as correlation without knowing whether the said random variables were centered), but arrived at the correct solution? $\endgroup$ – David Jul 25 at 0:47
  • $\begingroup$ Yes, surely a relevant premise is lacking in the text of the screenshot. Apart from this, the solution is correct. $\endgroup$ – Anatoly Jul 25 at 9:11
  • $\begingroup$ What approach did you use to know that the solution is correct btw? What I did was form the 3x3 covariance matrix, which is PSD. And then I found the determinant, which for a PSD matrix is $\geq 0$, and then solved for the correlation coefficient. $\endgroup$ – David Jul 25 at 18:45
  • $\begingroup$ Yes, this is the simplest way to find the solution. $\endgroup$ – Anatoly Jul 26 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.