10
$\begingroup$

I've read that the correlation coefficient between two random variables may be viewed as the cosine as the angle between them, but I can't find any solid explanation.

To be concrete, let $X$ and $Y$ be random variables on $(\Omega, \mathcal{F}, P)$ with correlation coefficient $\rho$. Assume $X,Y \in L^2(\Omega,\mathcal{F},P)$. The quantity $\rho$ is defined as $$ \rho := \frac{Cov(X,Y)}{\sqrt{Var(X)Var(y)}}. $$ Letting $\mu_X := E(X)$ and $\mu_Y := E(Y)$, note $$ Cov(X,Y) = E((X - \mu_X)(Y - \mu_Y)) = \left< X - \mu_X, Y - \mu_Y\right>_{L^2} $$ and $$ Var(X) = E((X - \mu_X)^2) = ||X - \mu_X||_{L^2}^2, $$ where $L^2 := L^2(\Omega, \mathcal{F}, P)$. Thus $$ \rho = \frac{\left< X - \mu_X, Y - \mu_Y\right>_{L^2}}{||X - \mu_X||_{L^2} ||Y - \mu_Y||_{L^2}}. \qquad (1) $$ Compare this with the Euclidean space inner product result that for two vectors $x,y \in \mathbb{R}^n$, $$ \cos(\theta) = \frac{x^Ty}{||x||\, ||y||}, $$ where $\theta$ is the angle between $x$ and $y$.

The recurring claim I read is that I can think of $\rho$ as the cosine of the "angle" between the two random variables, but it seems it only makes sense in terms of $L^2$ inner products. But in that case, the only notion of "angle" between to elements $X,Y \in L^2$ I can think of is $$ \cos(\theta) = \frac{\left< X,Y \right>_{L^2}}{||X||_{L^2} ||Y||_{L^2}}. \qquad (2) $$

So, two questions:

  1. Is Eqn. (2) a valid notion of angle in $L^2$?
  2. Are Eqns (1) and (2) somehow equlivalent?

If both of these are true, I can justify viewing $\rho = \cos(\theta)$.

$\endgroup$
1
  • 1
    $\begingroup$ $(2)$ measures angle between $X$ and $Y$. $(1)$ measures angle between $X-EX$ and $Y-EY$. The latter is much more meaningful in describing how $X$ and $Y$ vary together. For example, if $X,Y$ is uniform on $X+Y=2C$ and $|X-C|,|Y-C|<\epsilon$, $(1)$ will correctly yield $\rho = -1$ while $(2)$ will yield $cos(\theta)\approx 1$. $\endgroup$
    – A.S.
    Commented Nov 19, 2015 at 5:12

2 Answers 2

7
$\begingroup$

$\newcommand{\lowersub}[1]{_{\lower{0.5ex}{\small #1}}}$ The Euclidean norm of an $n$-dimensional vector $(X_1, X_2,...,X_n)^\top$ is ${\lVert x\rVert}_2 =\sum\limits_{k=1}^n {X}^2_k$, which is a measure of the distance from the origin.

In a similar way, $\sqrt{\mathsf E\Big(\!\big(X-\mathsf E(X)\big)^2\Big)}$ is a measure of the magnitude a random variable deviates from its mean.   Thus the standard deviation may be regarded as the weighted norm of the centred random variable.

( As A.S. commented, centering gives a more meaningful comparison of how two random variables deviate from their means. )

For discrete random variable we have: $$\lVert X-\mu\lowersub X\rVert = \sqrt{\sum_{\forall x} (x-\mu\lowersub{X})^2\mathsf P(X=x)}$$

And for a continuous real-valued random variable we have: $$\lVert X-\mu\lowersub{X}\rVert = \sqrt{\int_\Bbb R (x-\mu\lowersub{X})^2\,f\lowersub{X}(x)\operatorname d x}$$

Likewise, the angle between two $n$-dimensional vectors, $\vec x, \vec y$ is related to their inner product, and norms, by the cosine rule.

$$\cos \alpha\lowersub{\vec x,\vec y} = \frac{\langle \vec x, \vec y\rangle}{{\lVert \vec x\rVert}_2{\lVert \vec y\rVert}_2}$$

Similarly the co-variance, of two centered random variables, is analogous to an inner product, and so we have the concept of correlation as the cosine of an angle.

$$\rho\lowersub{X,Y}=\dfrac{\mathsf E((X-\mu\lowersub{X})(Y-\mu\lowersub{Y}))}{\sqrt{\mathsf E((X-\mu\lowersub{X})^2)\,\mathsf E((Y-\mu\lowersub{Y})^2)}}=\dfrac{\langle X-\mu\lowersub{X} , Y-\mu\lowersub{Y}\rangle}{\lVert X-\mu\lowersub{X}\rVert\,\lVert Y-\mu\lowersub{Y}\rVert} = \cos \theta\lowersub{X,Y}$$

$\endgroup$
5
  • $\begingroup$ I am confused by the notion of "inner product of random variables." I only understand inner product in context of random variables. When you right $\langle X, Y \rangle$ for random variables $X$ and $Y$, how do you actualy take the dot product between 2 random variables? $\endgroup$ Commented Jun 26, 2020 at 0:19
  • $\begingroup$ It is the expectation of the product. $\endgroup$ Commented Jun 26, 2020 at 0:24
  • $\begingroup$ Ohh that makes more sense now. If you have 2 vectors that are orthogonal, does that necessarily mean the correlation coefficient is zero? It seems it's not necessarily the case unless the mean of each vector is zero. But if the mean of each vector isn't zero, then it seems the inner product isn't guaranteed to be zero, and therefore, the correlation isn't guaranteed to be zero? $\endgroup$ Commented Jun 26, 2020 at 0:28
  • $\begingroup$ The (co)variance is centred around the mean(s). It's analogous to the inner product of the displacement vectors from a point. $\endgroup$ Commented Jun 26, 2020 at 3:02
  • $\begingroup$ I have a similar question. If you are looking at uncentered vectors $x$ and $y$, and they are orthogonal, that doesn't necessarily mean they are uncorrelated right? On second thought it seems my question doesn't make sense. In order for 2 vectors to form an angle and have the same origin, they need to be centered, right? Otherwise they wouldn't originate from the same point, so they can't form an angle. $\endgroup$
    – 24n8
    Commented Jul 11, 2020 at 18:38
3
$\begingroup$

$X-\mu_X$ is the projection of $X$ onto the plane $E=0$ in $L^2$. So $\rho$ measures the angle between the two projected vectors $X-\mu_X$, $Y-\mu_Y$. This has the effect that replacing $X$ by an "affine replica" $\alpha X+\beta$ resulting from a change of scales (think of $X$ being a temperature) does not change $\rho$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .