Measuring correlation of a truncated sample

Question

Suppose samples $(X_i,Y_i)$ were drawn from a multinomial distribution $N(\mu_X, \mu_Y, \sigma_X, \sigma_Y)$. The correlation between $X$ and $Y$ can be then estimated as $$\hat{\rho}_{XY}=\frac{1}{N}\sum_{i,j}\frac{(X_i - \mu_X)(Y_i - \mu_Y)}{\sigma_X\sigma_Y}$$ But now suppose tha all samples with $X_i>c$ are discarded, where $c$ is some constant. How does that change the correlation estimate?

The way I see it, the sum can be decomposed as $$\hat{\rho}_{XY}=\frac{1}{\sigma_X\sigma_Y}\sum_{j}(Y_i - \mu_Y)\left( \frac{1}{N_C}\sum_{i\in C}(X_i - \mu_X) + \frac{1}{N_{C'}}\sum_{i\in C'}(X_i - \mu_X)\right)$$ where $C=\lbrace i;X_i\leq c\rbrace$, $C=\lbrace i;X_i > c\rbrace$ and $N$, $N_{C'}$ is the number of elements in $C$ and $C'$, respectively. Now now the correlation changes depending on the sign of the term $\frac{1}{N_{C'}}\sum_{i\in C'}(X_i - \mu_X)$ which is discarded: if $c>\mu_X$ then all terms in that sum will be positive so throwing these away will reduce the correlation, and vice versa if $c<\mu_X$. Is there a more intuitive way to see this? Also, what would be some unbiased estimator for the underlying distribution (i.e. the distribution without the discarded samples), if $c$ is known?

Answer 1

Think of the scatterplot of the $(X_i,Y_i)$ pairs when the correlation is positive. When you remove $X_i$'s that exceed $C,$ they will be at the upper-right end of the 'ellipse', The $Y_i$ paired with these $X_i$'s will TEND to be at the same extreme. So some points at the 'top' end of both distributions will disappear, and these have relatively large 'leverage' in determining the positive correlation.

The argument for negative correlation is the same, but now it's points at the upper-left of the scatterplot that get discarded.

Here is a plot with $\rho = .8,$ and $c = 1.$ The red points are the ones omitted, resulting in a decrease of the correlation to $\rho_c \approx .69.$ (Relevant R code follows the plot.)

 m = 20000;  u = rnorm(m);  v = rnorm(m);  w = rnorm(m)
 x = u + 2*w;  y = v + 2*w
 cor(x, y)
 ## 0.8048806
 c = 1
 x.kept = x[x < c];  y.kept = y[x < c]
 cor(x.kept, y.kept)
 ## 0.6884493
 par(pty="s")  # square plotting area
  plot(x, y, col="red", pch=".")
  points(x.kept, y.kept, pch=".")
 par(pty="m")  # return to default plot