1
$\begingroup$

Let $n_{\mathrm{sample}}$ be a sample size and $n_{\mathrm{pop}}$ the population size. Generate an independent variable $x$ by picking $n_{\mathrm{sample}}$ numbers from the natural numbers from $1$ to $n_{\mathrm{pop}}$ without repetition. Generate a dependent variable $y$ by doing the same thing, picking $n_{\mathrm{sample}}$ numbers from the natural numbers from $1$ to $n_{\mathrm{pop}}$ without repetition. Then compute the linear regression $y = a + bx$.

This model is clearly symmetric in $x$ and $y$. Hence if one does this repeatedly for sufficiently large $n_{\mathrm{pop}}$ and $n_{\mathrm{sample}}$ one should expect that $a$ is on average $0$ and $b$ is on average $1$ (with some variation around this). I did this numerically and while a distribution of $a$ that is symmetric around $0$ looks credible, the average for $b$ is smaller than $1$. Here is some basic code in R that does this experiment.

n_pop  <- 100

n_sample <- 20

draws <- 1000

M <- matrix(c(1:(2*draws)), nrow=draws)

for (j in 1:draws) {

  x <- sort(sample(1:n_pop,n_sample,replace=FALSE))

  y <- sort(sample(1:n_pop,n_sample,replace=FALSE))

  xy_model <- lm(y~x)

  M[j,] <- coef(xy_model)

}

mean(M[,1])

mean(M[,2])

sd(M[,1])

sd(M[,2])

hist(M[,1],breaks=20)

hist(M[,2], breaks=20)

So what is going on? What is the expected value of the coefficient $b$ and what is its distribution?

edit: some numerical evidence that the coefficient $b$ is not $1$ (and $a$ is not zero either), change the number of draws to 10000 to get higher confidence (takes a few seconds to run on my computer) and add the two lines

t.test(M[,2], mu=1, conf.level=0.99)

t.test(M[,1], mu=0, conf.level=0.99)

gives a 99% confidence interval of $[0.9757861, 0.9828772]$ for $b$ and a p-value of $10^{-16}$ for $b=1$. Similarly the $a$ coefficient is around $1$ and almost certainly not zero.

I suspect this is related to the fact that ordinary least squares minimizes the squared distance in the $y$ direction which is not symmetric with respect to $x$ and $y$. A Deming regression which minimizes euclidean distance to the line should get the $a=0$ and $b=1$ that I expected.

$\endgroup$
4
  • $\begingroup$ Just to check your code have you tried running it when you set $y_i=x_i$ to see if this gives $b=1$ $\endgroup$
    – user121049
    Oct 17 '17 at 9:14
  • $\begingroup$ If you compute the linear model lm(x~x) you get x = 0 +1*x with zero error as expected. $\endgroup$
    – quarague
    Oct 17 '17 at 9:56
  • $\begingroup$ You say that the average for $b$ is smaller than $1$. Independently from the number of repetitions you do, you will see almost surely an estimated $\hat b\neq 1$. The simplest thing I can think of is to store all the estimated $\hat b$ and compute a $95\%$ confidence interval for that. If $1$ falls inside that interval then there is no statistical evidence for stating that $b$ is different from $1$. $\endgroup$
    – Ilis
    Oct 17 '17 at 12:14
  • $\begingroup$ See here to see how to do this test en.wikipedia.org/wiki/… $\endgroup$
    – user121049
    Oct 17 '17 at 12:42
0
$\begingroup$

I cross posted the same question here: https://stats.stackexchange.com/questions/308785/linear-regression-symmetry-of-model-does-not-lead-to-symmetry-of-coefficients

and got a nice answer with detailed explanation there.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.