# Measuring correlation of a truncated sample

Suppose samples $(X_i,Y_i)$ were drawn from a multinomial distribution $N(\mu_X, \mu_Y, \sigma_X, \sigma_Y)$. The correlation between $X$ and $Y$ can be then estimated as $$\hat{\rho}_{XY}=\frac{1}{N}\sum_{i,j}\frac{(X_i - \mu_X)(Y_i - \mu_Y)}{\sigma_X\sigma_Y}$$ But now suppose tha all samples with $X_i>c$ are discarded, where $c$ is some constant. How does that change the correlation estimate?

The way I see it, the sum can be decomposed as $$\hat{\rho}_{XY}=\frac{1}{\sigma_X\sigma_Y}\sum_{j}(Y_i - \mu_Y)\left( \frac{1}{N_C}\sum_{i\in C}(X_i - \mu_X) + \frac{1}{N_{C'}}\sum_{i\in C'}(X_i - \mu_X)\right)$$ where $C=\lbrace i;X_i\leq c\rbrace$, $C=\lbrace i;X_i > c\rbrace$ and $N$, $N_{C'}$ is the number of elements in $C$ and $C'$, respectively. Now now the correlation changes depending on the sign of the term $\frac{1}{N_{C'}}\sum_{i\in C'}(X_i - \mu_X)$ which is discarded: if $c>\mu_X$ then all terms in that sum will be positive so throwing these away will reduce the correlation, and vice versa if $c<\mu_X$. Is there a more intuitive way to see this? Also, what would be some unbiased estimator for the underlying distribution (i.e. the distribution without the discarded samples), if $c$ is known?

• Your second question about estimation is a bit vague. Which parameters do you want to estimate. Also, the best tactic may depend on $\rho$ and $c$. Give some more specifics and I (or someone else) will take a look. Jul 25, 2015 at 22:38
• To find the correlation after some points are discarded, you need to consider that the center of the data cloud is changed when points are discarded. Jul 25, 2015 at 22:45
• Indeed the sample mean and variance will change, but suppose we know what the true mean and variance of the generating distribution are, just not the correlation (hence why I put the population mean and variance in the correlation estimator, rather than their respective estimators which depend on $c$). Jul 26, 2015 at 12:14

Think of the scatterplot of the $(X_i,Y_i)$ pairs when the correlation is positive. When you remove $X_i$'s that exceed $C,$ they will be at the upper-right end of the 'ellipse', The $Y_i$ paired with these $X_i$'s will TEND to be at the same extreme. So some points at the 'top' end of both distributions will disappear, and these have relatively large 'leverage' in determining the positive correlation.

The argument for negative correlation is the same, but now it's points at the upper-left of the scatterplot that get discarded.

Here is a plot with $\rho = .8,$ and $c = 1.$ The red points are the ones omitted, resulting in a decrease of the correlation to $\rho_c \approx .69.$ (Relevant R code follows the plot.)

 m = 20000;  u = rnorm(m);  v = rnorm(m);  w = rnorm(m)
x = u + 2*w;  y = v + 2*w
cor(x, y)
## 0.8048806
c = 1
x.kept = x[x < c];  y.kept = y[x < c]
cor(x.kept, y.kept)
## 0.6884493
par(pty="s")  # square plotting area
plot(x, y, col="red", pch=".")
points(x.kept, y.kept, pch=".")

• Playing with your R code I see that the change in correlation does not depend on mean as I originally thought; the absolute value of the correlation is in fact reduced for any value of $c$ which makes sense in hindsight. Jul 26, 2015 at 12:12
• Incidentally, to generate samples with a known correlation rho I used instead rho = 0.8; u = rnorm(m); v = rnorm(m); x = u; y = rho * u + sqrt(1 - rho^2) * v; which follows from the Cholesky decomposition Jul 26, 2015 at 12:17
• @ScarletPumpernickel: Glad you played with the R code. I played a bit changing $c$. Of course, your method of getting correlated samples is better--and allows for easy changes in $\rho$. I didn't know your level of tolerance for R code, so picked the simplest method I could imagine. Jul 26, 2015 at 14:59