# Correlation coefficient and regression line : Geometric intuition

correlation coefficient

$$r = \frac{1}{n}\sum_{i=1}^n\frac{(x_i-\bar x)(y_i-\bar y)}{\sigma_x\cdot\sigma_y}$$

may be thought of as cosine of angle between two $n$-dimensional vectors

$$(x_1- \bar x, x_2- \bar x,\ldots, x_n- \bar x) \text{ and } (y_1- \bar y,y_2- \bar y,\ldots,y_n- \bar y)$$

1. But what is special about these two vectors?

why don't we take take angles between any other two vectors?

Yes, I know the intuition behind the algebra,that we subtract $\bar x\text{ and } \bar y$ so that the mean is zero and the the sign of products gives us the correlation and we divide by $\sigma_x\cdot\sigma_y$ to remove the effects of scaling of the distributions.

I want to know the geometric intuition in terms of angle between two vectors.

1. Also I would like to know the geometric intuition behind the relationship

slope of regression line $$=r \cdot \frac{\sigma_y}{\sigma_x}$$

I know that when $r = 1,$ the slope of regression line should be $\frac{\sigma_y}{\sigma_x}$

What I don't understand is how the cosine of angle between two vectors

$$(x_1- \bar x, x_2- \bar x,\ldots,x_n- \bar x) \text{ and } (y_1- \bar y,y_2- \bar y,\ldots,y_n- \bar y)$$

when multiplied to $\frac{\sigma_y}{\sigma_x}$ gives us the slope.

I understand your pain in self teaching as I am doing the same. I assume you know trignometry. Imagine two vectors as below and $$\theta$$ as angle between them

$$\vec{a} = a_1\hat{i} + a_2\hat{j} \\ \vec{b} = b_1\hat{i} + b_2\hat{j} \tag{1}$$

Using law of cosines it can be proven that,

$$\text{cos }\theta = \dfrac{\vec{a}\bullet\vec{b}}{\lVert a \rVert \lVert b \rVert} \tag{2}$$

where

$$\vec{a}\bullet\vec{b} = a_1b_1 + a_2b_2 \tag{3}$$

The vector form can also be expressed in matrix multiplication as below. $$\vec{a} \bullet \vec{b} = \begin{matrix} \begin{bmatrix} a_1 & a_2 \end{bmatrix} \\[2.8ex] \end{matrix} \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} = \begin{bmatrix} a_1 \\ a_2 \end{bmatrix} \bullet \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} = a_1b_1 + a_2b_2 = \sum_i^2 a_i b_i \tag{4}$$

So essentially if you have sample set $$(X,Y) = \{ (x_1,y_1), (x_2,y_2) \}$$, you could visualize 2D vectors constructed out of this sample set $$(X,Y)$$.

Let the 2D vectors be $$\vec{x} = x_1\hat{i} + x_2\hat{j} \\ \vec{y} = y_1\hat{i} + y_2\hat{j} \tag{5} \\$$

$$\text{cos }\theta = \dfrac{\vec{x}\bullet\vec{y}}{\lVert x \rVert \lVert y \rVert} \tag{6}$$

where

$$\vec{x}\bullet\vec{y} = x_1y_1 + x_2y_2 \tag{7}$$

In matrix multiplication form,

$$\vec{x} \bullet \vec{y} = \begin{matrix} \begin{bmatrix} x_1 & x_2 \end{bmatrix} \\[2.8ex] \end{matrix} \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \bullet \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = x_1y_1 + x_2y_2 = \sum_i^2 x_i y_i \tag{8}$$

Now if you center the sample set, that is subtract each sample point from its mean, still the law of cosine could be applied, but with now a slightly different vector.

Let $$\vec{x_c} = (x_1 - \overline{x})\hat{i} + (x_2 - \overline{x})\hat{j} \\ \vec{y_c} = (y_1 - \overline{y})\hat{i} + (y_2 - \overline{y})\hat{j} \tag{9}$$

Then using same steps as above, we can show that,

$$\text{cos }\theta = \dfrac{\vec{x_c}\bullet\vec{y_c}}{\lVert x_c \rVert \lVert y_c \rVert} \tag{10}$$

And the dot product,

$$\vec{x_c} \bullet \vec{y_c} = \begin{matrix} \begin{bmatrix} x_1-\overline{x} & x_2 -\overline{x} \end{bmatrix} \\[2.8ex] \end{matrix} \begin{bmatrix} y_1 -\overline{y} \\ y_2 -\overline{y} \end{bmatrix} = \begin{bmatrix} x_1 -\overline{x} \\ x_2 -\overline{x} \end{bmatrix} \bullet \begin{bmatrix} y_1 -\overline{y} \\ y_2 -\overline{y} \end{bmatrix} = \sum_i^2 (x_i - \overline{x}) (y_i - \overline{y}) \tag{11}$$

and modulus of the vectors of course,

$$\lVert x_c \rVert = \sqrt{\sum_i^2 (x_i - \overline{x})^2} \\ \lVert y_c \rVert = \sqrt{\sum_i^2 (y_i - \overline{y})^2} \tag{12} \\$$

Thus eq.(10) becomes,

$$\text{cos }\theta = \dfrac{\vec{x_c}\bullet\vec{y_c}}{\lVert x_c \rVert \lVert y_c \rVert} = \dfrac{\sum_i^2 (x_i - \overline{x}) (y_i - \overline{y})}{\sqrt{\sum_i^2 (x_i - \overline{x})^2}\sqrt{\sum_i^2 (y_i - \overline{y})^2}} = \dfrac{\text{cov}(X,Y)}{s_Xs_Y} = r \tag{13}$$

where $$\text{cov}(X,Y)$$ is sample covariance and $$s_X,s_Y$$ are unbiased sample standard deviations.

This is how the $$r$$ and angle between the vectors constructed out of the sample set are related. I have constructed with only sample size of 2 pairs $$(x_1, y_1), (x_2,y_2)$$, but this could be extended to any number of sample set size.

Thus a N sized sample set, could be imagined as 2 vectors of N dimensions and of course we cannot visualize it. But in that unimaginable N dimension, the angle $$\theta$$ between those "vectors" still be on a plane (2D), so the law of cosine still applies. Below is an image, where for $$N=3$$, we could visualize 2, 3D vectors, and you can see the angle sweeped between them is still on a 2D plane. Of course we cannot go beyond 3D, but you can get the point.

Note that, the value of cosine ranges between $$\pm 1$$. So when both vectors are in same direction, the $$\theta$$ is 0, thus cos$$\theta$$ = 1, maximum value indicating perfect linearity. Similarly when both vectors are in opposite direction, $$\theta = 180^{\circ}$$, implying cos$$\theta$$ = -1. When the vectors are perpendicular to each other, $$\theta = 90^{\circ}$$ implying cos$$\theta = 0$$, thus zero correlation.

$$\left( \frac{Y - \nu} \tau \right) = \rho \cdot \left( \frac{X - \mu} \sigma \right)$$ The two quantities in $\Big( \text{parentheses}\Big),$ or "round brackets" or whatever your preferred term for those things is, are "z-scores". A z-score is how many standard deviations above average something is. (And a negative number of S.D.s above average means below average.) If $X$ is a certain number of S.D.'s above average, then $Y$ is above or below average according as the correlation $\rho$ is positive or negative. With perfect positive correlation $\rho=1,$ if $X$ is a certain number of S.D.s above average, then $Y$ is the same number of S.D.s above average. With perfect negative correlation $\rho=-1,$ if $X$ is a certain number of S.D.s above average, then $Y$ is the same number of S.D.s below average. With zero correlation, $\rho=0,$ if $X$ has any value at all, then $Y$ has the average $Y$-value. With correlation $\rho= 1/2,$ If $X$ is a certain number of S.D.s above average, then $Y$ is half that number of S.D.s above average. And so on.

Here of course all of this is with $(X,Y)$ on the line. The $Y$ value given by this equation is the average $Y$-value for a given $X$-value. In case it's based on least-squares estimation with a finite random sample from a population, it's the estimated average $Y$-value for a given $X$-value.

The standard deviation is the average of the squares of the deviations. The deviations are $x_i-\overline x$ and $y_i-\overline y.$

• Thanks Mr. Hardy for your answer. It seems I was not very clear with my question(english is not my first language). I wanted to understand how r relates to angle between two vectors. I know geometric intuition of r. Also, $$\left( \frac{Y - \nu} \tau \right) = \rho \cdot \left( \frac{X - \mu} \sigma \right)$$ but average of all $$\left( \frac{Y - \nu} \tau \right) \cdot \left( \frac{X - \mu} \sigma \right)$$ is equal to $\rho$ , another non intuitive relation. – q126y Sep 3 '18 at 5:05
• sorry, I meant, I know algebraic intuition of r in my last comment. – q126y Sep 3 '18 at 8:21
• One thing $\rho$ has to do with angles is this. Suppose the correlation between $X$ and $Y$ is $0.97,$ and that between $Y$ and $Z$ is $0.98.$ Then one angle is $\arccos0.97\approx14.069877476^\circ$ (if I can believe my software) and the other is $\arccos0.98\approx11.47834095\circ,$ so the correlation between $X$ and $Z$ cannot be less than about $\cos((14.069877476+11.47834095)^\circ)$ nor more than about $\cos((14.069877476-11.47834095)^\circ). \qquad$ – Michael Hardy Sep 3 '18 at 18:16
• Thanks Mr. Hardy, that relationship was not known to me. However I was looking for intuition behind r as cosine of angle between two vectors. How do we get the vectors geometrically( I know how do we get it algebraically). $$(x_1- \bar x, x_2- \bar x,\ldots, x_n- \bar x) \text{ and } (y_1- \bar y,y_2- \bar y,\ldots,y_n- \bar y)$$ – q126y Sep 4 '18 at 3:35
• @q126y : Precisely which parts you understand and which you don't is not fully clear to me. Would it help to point out that the mapping $(x_1,\ldots,x_n) \mapsto (x_1-\overline x, \ldots, x_n - \overline x)$ is an orthogonal projection onto a subspace of dimension $n-1 \text{?} \qquad$ – Michael Hardy Sep 5 '18 at 2:12