3
$\begingroup$

I was reading on wikipedia about the pearson correlation coefficient. Assuming the data has zero mean it can be written as

$$ \rho = \frac{ \sum x_i y_i } {\sqrt{\sum x_i^2 \sum y_i^2}} $$

The caption below this image says:

[...] Note that the correlation reflects the non-linearity and direction of a linear relationship (top row), but not the slope of that relationship (middle), nor many aspects of nonlinear relationships (bottom). [...]

(bold text emphasis added by me)

The middle row of the picture shows several distributions that are perfectly correlated ($\rho=1$) and illustrates, that in that case the correlation coefficient does not change when the slope changes (apart from the case if either $x$ or $y$ is constant).

However, I have my doubts whether the correlation coefficient is really independent of the slope, when the correlation is not perfect (ie $\rho<1$). In other words, how does the correlation coefficient change, when I apply a simple rotation

$$ x'_i = x_i \cos(\alpha) + y_i \sin(\alpha) \\ y'_i = -x_i \sin(\alpha) + y_i \cos(\alpha) $$

to the data?

Note that the rotation does not change the mean values if $\sum x = \sum y = 0$, but even in the simple form as written above I didn't manage to derive an expression for

$$ \rho(\alpha) = ?? $$

yet. Or maybe I am just a bit confused and the correlation coefficient really does not change....

$\endgroup$
3
$\begingroup$

I suppose that $\sum_i x_i = \sum_i = y_i = 0$. Moreover, $P_x = \sum_i x_i^2$, $P_y = \sum_i y_i^2$ and $C_{xy} = \sum_i x_i y_i$.

Then, the sample Pearson coefficient $\rho$ based on data $x_i$ and $y_i$ produced by random variables $X$ and $Y$ is:

$$\rho = \frac{C_{xy}}{\sqrt{P_x P_y}}.$$

Notice that:

$$P_{x'} = \sum_i {x'}^2_i = \sum_i (x_i \cos \alpha + y_i \sin \alpha)^2 = \\ \cos^2 \alpha\sum_i x_i^2 + \sin^2 \alpha\sum_i y_i^2 + 2\sum_i x_i y_i \sin \alpha \cos \alpha = \\\cos^2 \alpha P_x + \sin^2 \alpha P_y + \sin(2\alpha) C_{xy},$$

$$P_{y'} = \sum_i {y'}^2_i = \sum_i (-x_i \sin \alpha + y_i \cos \alpha)^2 = \\ \sin^2 \alpha\sum_i x_i^2 + \cos^2 \alpha\sum_i y_i^2 - 2\sum_i x_i y_i \sin \alpha \cos \alpha = \\\sin^2 \alpha P_x + \cos^2 \alpha P_y - \sin(2\alpha) C_{xy},$$

and

$$C_{x'y'} = \sum_i x_i' y_i' = \sum_i (x_i\cos \alpha + y_i \sin \alpha)( -x_i \sin \alpha + y_i \cos \alpha) = \\ -\sum_i x_i^2\sin\alpha\cos \alpha + \sum_i x_i y_i (\cos^2 \alpha - \sin^2 \alpha) + \sum_i y_i^2\sin\alpha\cos \alpha = \\ \frac{1}{2}\sin(2\alpha)(P_y - P_x) + C_{xy} \cos(2\alpha).$$

Consider $\alpha = \frac{\pi}{2}$ and join all pieces togheter:

$$\rho' = \frac{C_{x'y'}}{\sqrt{P_{x'} P_{y'}}} = \frac{\frac{1}{2}\sin(2\frac{\pi}{2})(P_y - P_x) + C_{xy} \cos(2\frac{\pi}{2})}{\sqrt{(\cos^2 \frac{\pi}{2} P_x + \sin^2 \frac{\pi}{2} P_y + \sin(2\frac{\pi}{2}) C_{xy})(\sin^2 \frac{\pi}{2} P_x + \cos^2 \frac{\pi}{2} P_y - \sin(2\frac{\pi}{2}) C_{xy})}} = \\ = \frac{-C_{xy}}{\sqrt{P_yP_x}} = - \rho. $$

Conclusion: rotation affects Peason coefficient.

Addition

In general, the new Pearson coefficient, as a function of $\alpha$, is

$$\rho' = \frac{C_{x'y'}}{\sqrt{P_{x'} P_{y'}}} = \frac{\frac{1}{2}\sin(2\alpha)(P_y - P_x) + C_{xy} \cos(2\alpha)}{\sqrt{(\cos^2 \alpha P_x + \sin^2 \alpha P_y + \sin(2\alpha) C_{xy})(\sin^2 \alpha P_x + \cos^2 \alpha P_y - \sin(2\alpha) C_{xy})}}. $$

$\endgroup$
  • $\begingroup$ Sorry, I should have been more clear in my question. A change from $\rho=1$ to $\rho=-1$ is also illustrated in the wikipedia image, thus your example is not the most convincing one. On the other hand I guess by simply using a different value for $\alpha$ one can show that also the magnitude of $\rho$ is changing. Thanks anyhow $\endgroup$ – formerlyknownas_463035818 May 30 '16 at 10:47
  • $\begingroup$ Actually now I am a bit confused. I have an application where I need to maximize the correlation between some $X$ and $Y + \beta Z$ depending on $\beta$ (see e.g. here). The "slope of that relationship" is irrelevant and can change. But now if $\rho$ changes with that slope I am not sure if I still find the correct maximum. It could well be that the maximum is independent of rotations, but that would be a new question i guess.... could you suggest me some reading or do you have a tip to clarify my confusion? $\endgroup$ – formerlyknownas_463035818 May 30 '16 at 11:34
  • $\begingroup$ Don't have some reference. But you can find the new Pearson coefficient using some algebra similar to mine $\endgroup$ – the_candyman May 30 '16 at 11:39
  • 2
    $\begingroup$ @tobi303: Maybe the focus on "slope (non-)invariance" is confusing you. Correlation is not invariant under rotation, but it is invariant under scaling. So changing the slope by scaling does not affect the correlation but changing the slope by rotating does affect the correlation. $\endgroup$ – citronrose May 30 '16 at 11:55
  • 1
    $\begingroup$ @citronrose you are right and i was just thinking too complicated. For the application i mentioned only scaling matters but no rotations. Nevertheless i find the effect of rotations quite interesting $\endgroup$ – formerlyknownas_463035818 May 30 '16 at 12:25
1
$\begingroup$

I think there are big absents: the variance matrices which are, in all these questions, the central concept.

Using matrix-vector notations (instead of all-algebraic calculations), and assuming that we work on centered data, we have the following transformation:

$$X'=RX \ \ \text{with} \ \ R=\begin{bmatrix}\cos(\alpha)&-\sin(\alpha)\\\sin(\alpha)&\cos(\alpha)\end{bmatrix} \ \ \text{and}$$ $$X'=\begin{bmatrix}x'_1&x'_2&\cdots&x'_n\\y'_1&y'_2&\cdots&y'_n\end{bmatrix}, \ X=\begin{bmatrix}x_1&x_2&\cdots&x_n\\y_1&y_2&\cdots&y_n\end{bmatrix}$$

Thus $$X'X'^T=R(XX^T)R^T$$

In other words $$V'=RVR^T$$

(see "A more general identity" in this). by naming $V$ and $V'$, resp. the old and new (co)variance matrices.

which is the way (co)variance matrices are modified (a kind of generalisation of the property $var(aX)=a^2 var(X)$). Thus the new variance matrix is very different from the old variance matrix, even if one normalize each one in order to reason on correlation matrices instead of variance matrices.

The old formula $\rho=\dfrac{V_{12}}{\sqrt{V_{11}V_{22}}}$ is replaced by the new one : $\rho'=\dfrac{V'_{12}}{\sqrt{V'_{11}V'_{22}}}$

(see the final result of @the_candyman).

$\endgroup$
  • $\begingroup$ sorry, I dont really understand your answer. I wouldn't say that the variance matrices are absent in my question or in the other answer, they are just written in a slightly different way. $\endgroup$ – formerlyknownas_463035818 May 30 '16 at 11:45
  • 1
    $\begingroup$ It's all right, they were implicitly there. I just spent a little time re-writing things in a more readable way, in my opinion. $\endgroup$ – Jean Marie May 30 '16 at 11:50
  • $\begingroup$ ah ok got it. I actually prefer the "element-wise" notation to the matrix notation most of the time, but thats just my personal preference $\endgroup$ – formerlyknownas_463035818 May 30 '16 at 11:53
  • $\begingroup$ I understand this preference. It takes a long time to be confident into all-matrix-vector notations. Myself, there are cases, such as derivation with respect to a matrix, where I check my computations element-wise. $\endgroup$ – Jean Marie May 30 '16 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.