28
$\begingroup$

For building a recommendation system, I also use the Pearson correlation coefficient. This is the definition:

$r(x, y)=\frac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^n (x_i-\bar{x})^2 \cdot \sum_{i=1}^n (y_i-\bar{y})^2}}$

$x$ and $y$ are part of $\mathbb{R}$.

Now for coding, it is important to take care of all potential outcomes. For example, if the denominator is zero, you will have to filter that or throw an exception.

I came up with some arguments, one of them being that if all values of $x_i$ and/or $y_i$ were equal to the average of $x$ and/or $y$, then the denominator would be zero.

But how can I prove that the coefficient is either undefined (zero denominator) or in between -1 and 1? What is the best approach?

$\endgroup$
1
  • 3
    $\begingroup$ It is the Schwartz Inequality (but I keep misspelling his name). Depending where you are from, you might call it Cauchy Schwartz, or throw in Bunyakovsky, maybe others. $\endgroup$ Nov 13 '13 at 0:50
38
$\begingroup$

First of all Pearson's correlation coefficient is bounded between -1 and 1, not 0 and one. It's absolute value is bounded between 0 and 1, and that useful later.

Pearson's correlation coefficient is simply this ratio:

$$\rho = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$$

Both of the variances are non-negative by definition, so the denominator is $\ge 0$. The only way a singularity can occur is if one of the variables has 0 variance.

If two random variables are perfectly uncorrelated, (i.e. independent) then their covariance is 0. So 0 is a valid lower bound for the absolute value of the expression.

This can be shown like so:

$$Cov(X,Y) = E[(X-\bar{X})(Y-\bar{Y})] = E[XY] - E[X]E[Y]$$

if two random variables are independent, then $E[XY]=E[X]E[Y]$, and

$$Cov(X,Y) = E[XY] - E[X]E[Y] = E[X]E[Y] - E[X]E[Y] = 0.$$

Now for the upper bound. Here we apply the Cauchy-Schwarz inequality.

$$|Cov(X,Y)|^2 \le Var(X)Var(Y)$$

$$\therefore |Cov(X,Y)| \le \sqrt{Var(X)Var(Y)}$$

plug this result from the Cauchy-Schwarz inequality into the formula for $\rho$, and we get:

$$|\rho| = \left|\frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}\right| \le \frac{\sqrt{Var(X)Var(Y)}}{\sqrt{Var(X)Var(Y)}} = 1$$

Thus we have the absolute value of the correlation is bounded below by 0 and above by 1.

$\endgroup$
3
  • 3
    $\begingroup$ I don't think it's adequate to just quote the Cauchy-Schwarz inequality and say we're done; you need to actually show that expectation values satisfy the definition of an inner product space, which is non-obvious (or at least non-obvious to anyone who would need to ask the question in this thread) $\endgroup$
    – xdavidliu
    Apr 21 at 2:56
  • $\begingroup$ @xdavidliu Wikipedia's page on Cauchy-Schwarz offers a similar proof that explicitly say that expectation of a product of two random variables is an inner product. For those confused, just search on Google what operations quality as an inner product, and you will get a list of properties - not super intuitive but at least informative. $\endgroup$
    – Wiza
    Nov 14 at 0:21
  • $\begingroup$ To be fair, your don’t actually need the fact that Cauchy-Schwartz is proved in general for inner product spaces. You only need the implication of that for the covariance and product of variances of two random variables. That is relatively easy to prove, and to be honest the actual definition given of CS in many introductory mathematical statistics texts (as I was in when I wrote this 8 years ago…) $\endgroup$ Nov 16 at 5:04
2
$\begingroup$

Saw this while looking around. To complete above proof, $$ 0\leq Var(\frac{X}{\sigma_x}\pm\frac{Y}{\sigma_y}) = Var(\frac{X}{\sigma_x}) + Var(\frac{Y}{\sigma_y}) \pm 2Cov(\frac{X}{\sigma_x},\frac{Y}{\sigma_y}) $$

where (for both $X$ and $Y$) due to $Var(aX)=a^2Var(X)$ $$Var(\frac{X}{\sigma_x}) = \frac{1}{\sigma_x^2}Var(X)=\frac{\sigma_x^2}{\sigma_x^2}=1$$

and due to $Cov(aX,bY)=abCov(X,Y)$ $$Cov(\frac{X}{\sigma_x},\frac{Y}{\sigma_y}) = \frac{1}{\sigma_x\sigma_y}Cov(X,Y) = Corr(X,Y)$$

Hence $$ Var(\frac{X}{\sigma_x}\pm\frac{Y}{\sigma_y}) = 1+1\pm2Corr(X,Y)$$

$\endgroup$
1
$\begingroup$

Here's a standalone proof that should be much easier to understand and justify than quoting Cauchy-Schwarz. It's taken from sec 7.4 of Sheldon Ross's A First Course in Probability 10th edition, which I highly recommend.

Let $X$ and $Y$ be random variables with respective variances $\mathrm{Var}(X) = \sigma_x^2$ and $\mathrm{Var}(Y) = \sigma_y^2$. We then have

$$ 0 \leq \mathrm{Var} \left( \frac{X}{\sigma_x} \pm \frac{Y}{\sigma_y} \right) = 2 \pm 2 \mathrm{Corr} (X,Y). $$

From which it immediately follows that $-1 \leq \mathrm{Corr} (X,Y) \leq 1$.

$\endgroup$
4
  • 1
    $\begingroup$ You would need to prove that that expression is true $\endgroup$ May 17 at 3:39
  • $\begingroup$ which expression do you mean? The $\mathrm{Var} = 2 \pm 2 \mathrm{Corr}$ expression? I think that's pretty straightforward $\endgroup$
    – xdavidliu
    May 17 at 12:56
  • $\begingroup$ Not everyone thinks that. A proof would improve your answer. $\endgroup$ May 18 at 2:17
  • $\begingroup$ Variance of linear combinations of random variables can easily be found elsewhere, I'm not going to add that here, sorry. $\endgroup$
    – xdavidliu
    May 18 at 2:58
0
$\begingroup$

Let $\vec{a} = (x_1-\bar{x}, x_2-\bar{x}, \ldots, x_n-\bar{x})$ and $\vec{b}=(y_1-\bar{y},\ldots,y_n-\bar{y})$. Then your formula is just $\frac{\vec{a} \cdot \vec{b}}{\|\vec{a}\| \|\vec{b}\|}$. Since $\vec{a} \cdot \vec{b} = \|\vec{a}\| \|\vec{b}\| \cos(\theta)$ your formula reduces to $\cos(\theta)$, where $\theta$ is the angle between $\vec{a}$ and $\vec{b}$ in $n$-dimensional euclidean space.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.