10
$\begingroup$

Let $n$ be a fixed positive integer. Generate $n$ numbers $x_1, x_2, ..., x_n$ from the set $[0,1]$, with the probability distribution being the uniform one and the $x_i$ all being independent of each other. Now repeat this process to generate $y_1, ..., y_n$. If we let $X$ be a random variable which takes on $x_1, ..., x_n$ with probability $\frac{1}{n}$ each, and let $Y$ be a random variable which takes on $y_i$ whenever $X$ takes on a value of $x_i$. We can then compute the square of the correlation $R^2$ between $X$ and $Y$. What is the expected value of this $R^2$?

Another less rigorous phrasing of the problem is this: suppose we throw $n$ points at random on a graph spanning $[0,1] \times [0,1]$. What is the expected value of the $R^2$ of the line of best fit?

For instance, for $n=2$ the expected value is $1$ due to the $R^2$ value always being $1$. For $n=3$ one can numerically compute the expected value to be $\frac{1}{2}$. In general, it seems that the answer is $\frac{1}{n-1}$. I don't really have any idea how to do this problem in general; and even specific cases look nontrivial. Does anyone have any ideas? This looks like what should be a well-known result, but my searching didn't pick up on anything which looked useful.

This has applications in that when one is working with variables which are not expected to be very highly correlated, it is often difficult to tell when an $R^2$ value is significant. This result gives an idea of how big the $R^2$ needs to be for one to deduce there is some nontrivial correlation between two variables.

$\endgroup$
  • $\begingroup$ Your question seems to have drifted out of the scope of peoples attention. I've been thinking about it however, and may have a response for you... eventually $\endgroup$ – jameselmore Jan 22 '15 at 14:53
  • $\begingroup$ I found something related : mathforum.org/kb/message.jspa?messageID=7021085#reply-tree Unfortunately it does not leave any reference or proof, and it also uses normal distributions rather than uniform distributions. $\endgroup$ – dinoboy Jan 22 '15 at 18:30
  • $\begingroup$ I don't see how your initial phrasing of the question corresponds to the less rigourous one. Aren't $X, Y$ 1-dimensional random variables, uniform on $[0, 1]$? $\endgroup$ – Kimball Jan 25 '15 at 16:21
  • $\begingroup$ Yes, they are. But $X,Y$ act as the $x$ and $y$ coordinates of the ten points, as if $X$ takes on the value of $x_i$ then $Y$ is forced to take on the value of $y_i$. $\endgroup$ – dinoboy Jan 25 '15 at 18:15
  • $\begingroup$ But $y_i$ is independent of $x_i$, so $Y$ is independent of $X$. I think you need to reformulate the precise version of your question, for which the answer does not depend on $n$. $\endgroup$ – Kimball Jan 26 '15 at 16:43
6
+100
$\begingroup$

This problem seems simple...but its not. For example, see here for a rather complex analysis for the prima facie simple case of ratios of normal rv and ratios of sums of uniforms.

In general, if your pairs are not from a bivariate gaussian, there is no nice formula for $E[R^2]$.

Note:

$$R_n=\frac{n\sum x_iy_i-\sum x_i\sum y_i}{n^2s_Xs_Y}$$

This mess will have some distribution $f_{R_n}(r)$ that will be very sensitive to $n$.

I think your best bet is to simulate this (Monte Carlo) for $n\in [2....N]$ using a large number of trials (you can check convergence by running each simulation twice, with randomly chosen seeds and comparing these results to each other and to results from $n-1$).

Once you have this data, you can fit a curve to the it or some transformation thereof. Your general equation looks reasonable in terms of how the curve will look, since:

$$E[R^2_n] \xrightarrow{p} 0$$ for correlations between independent variables

Possible Solution

Since your variables are independent, I realized that we are really looking for the variance of the sample correlation (i.e., the square of the expected value of the standard error of the correlation coefficient (see p.6):

$$se_{R_n}=\sqrt{\frac{1-R^2}{n-2}}$$. However, you already know the true value of $R^2$, so you can increase the df in the denominator to get:

But: $R^2=0$ for independent variables, so it reduces to:

$$(se_{R_n})^2=\sigma^2_{R_n}=E[R^2_n]=\frac{1}{n-1}$$

There you have it...it matches your empirical results. As per Wolfies, I should note that this is an asymptotic result, but sums of uniform RVs generally exhibit good convergence properties ala CLT, so this may explain the good fit.

For further reading, see @soakley's nice reference. I was able to pull the relevant page from JSTOR:

enter image description here

or, if you're really motivated, you can get this recent article (2005) about your exact problem.

$\endgroup$
  • $\begingroup$ I think the difficulty arises from the sample standard deviations in the denominator. Would it suffice to estimate the covariance? It, like the correlation will tend to $0$, further, the standard deviations should both converge to $\frac{1}{2\sqrt3}$ $\endgroup$ – jameselmore Jan 29 '15 at 14:28
  • $\begingroup$ @jameselmore unfortunately, no, as the variability in the denominator will skew the expected value of $R^2$ relative to sample covariance$Cov(x,y)$, per jensens inequality. $\endgroup$ – user76844 Jan 29 '15 at 14:53
  • $\begingroup$ I completely agree that in general the problem is intractable. However in this case the answer turns out to be $\frac{1}{n-1}$, which hints that it could be possible. $\endgroup$ – dinoboy Jan 29 '15 at 17:40
  • $\begingroup$ @dinoboy I think you mean the answer appears to be $\frac{1}{n-1}$. A lot of grotesque functions look like reciprocal functions. So its not an indication that the solution is simple. The uniform distributions are actually not very nice, analytically. My proposal applies to your case in particular. $\endgroup$ – user76844 Jan 29 '15 at 18:06
  • $\begingroup$ Fair point. However, I've already done a large amount of numerical simulations and it appears to be very close to $\frac{1}{n-1}$ for $n$ ranging up to several hundred, and my simulations do not consistently generate results either below nor above $\frac{1}{n-1}$. While it could be true that $E[R_n^2] - \frac{1}{n-1}$ could be some pathological function which continually bounces above and below $0$, without concrete evidence that this is the case (i.e. an explicit computation for $n=3$) there's no reason to give up on the problem yet. $\endgroup$ – dinoboy Jan 29 '15 at 19:07
4
$\begingroup$

According to Kendall's Advanced Theory of Statistics (Exercise 16.17 in the 5th edition of Volume 1), Pitman (1937) showed the sample correlation coefficient $r$ has zero mean and variance or second moment of $$\sigma^2_{r}=E[r^2] = {1 \over {n-1}}$$ for any sample of size $n$ where $x$ and $y$ are independent continuous variates.

Checking the reference, we find he shows $r^2$ has an approximate $\mathrm{Beta} \left( {1 \over 2}, {{n-2} \over {2}}\right)$ distribution.

Reference: Pitman, E.J.G.. Significance tests which may be applied to samples from any population., v. 4, No. 1, II. The correlation coefficient test., v. 4, No. 2, $\it{Supp. J.R. Statist. Soc.},$ 1937.

$\endgroup$
  • $\begingroup$ Perhaps I'm just being dumb, but I don't think this result is actually what I want. I have Pitman's paper in front of me and the problem he discusses is given numbers $x_1, x_2, ..., x_n, y_1, ..., y_n$, consider the $n!$ possible pairings of the $x_i$ and $y_i$. Then he discusses the statistics of $r$ under the $n!$ possible permutations. While I wouldn't be surprised if the problem is equivalent, it doesn't seem at all obvious that it should be. $\endgroup$ – dinoboy Jan 30 '15 at 20:34
  • $\begingroup$ @dinoboy note that Pittmans approach implies x and y are independent, hence each x,y pair has equal probablity, and it reduces to a counting problem....its what you want. $\endgroup$ – user76844 Jan 30 '15 at 20:58
1
$\begingroup$

I'm just copying the section from

http://en.wikipedia.org/wiki/Coefficient_of_determination

I think it is what you are looking for.

A data set has n values marked $y_1...y_n$ (collectively known as $y_i$), each associated with a predicted (or modeled) value $f_1...f_n$ (known as $f_i$, or sometimes $ŷ_i$).

If $\bar{y}$ is the mean of the observed data:

$\bar{y}=\frac{1}{n}\sum_{i=1}^n y_i $ then the variability of the data set can be measured using three sums of squares formulas:

The total sum of squares (proportional to the variance of the data): $SS_\text{tot}=\sum_i (y_i-\bar{y})^2,$ The regression sum of squares, also called the explained sum of squares: $SS_\text{reg}=\sum_i (f_i -\bar{y})^2,$ The sum of squares of residuals, also called the residual sum of squares: $SS_\text{res}=\sum_i (y_i - f_i)^2\,$ The notations $SS_\text{R}$ and $SS_\text{E}$ should be avoided, since in some texts their meaning is reversed to Residual sum of squares and Explained sum of squares, respectively.

The most general definition of the coefficient of determination is

$R^2 \equiv 1 - {SS_{\rm res}\over SS_{\rm tot}}.$

Note: I can't tell from the preview if it looks ok. I'll keep trying to make it look ok, or just follow the link.

If nothing else, look at the inset figure to the right.

Here is the link to the graphic, with squares of data versus (difference of squared) $\bar{y}$ on the left compared to squares of data versus (difference of squared) fit line on right.

http://en.wikipedia.org/wiki/Coefficient_of_determination#mediaviewer/File:Coefficient_of_Determination.svg

$\endgroup$
  • $\begingroup$ jameselmore thanks a ton. I'll be trying to come up to speed on mathjax. $\endgroup$ – Paul Magnussen Jan 27 '15 at 19:25
  • $\begingroup$ You had it pretty close! Just need to use the $ operators $\endgroup$ – jameselmore Jan 27 '15 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.