# linear regression, symmetry of model does not lead to symmetry of coefficients

Let $n_{\mathrm{sample}}$ be a sample size and $n_{\mathrm{pop}}$ the population size. Generate an independent variable $x$ by picking $n_{\mathrm{sample}}$ numbers from the natural numbers from $1$ to $n_{\mathrm{pop}}$ without repetition. Generate a dependent variable $y$ by doing the same thing, picking $n_{\mathrm{sample}}$ numbers from the natural numbers from $1$ to $n_{\mathrm{pop}}$ without repetition. Then compute the linear regression $y = a + bx$.

This model is clearly symmetric in $x$ and $y$. Hence if one does this repeatedly for sufficiently large $n_{\mathrm{pop}}$ and $n_{\mathrm{sample}}$ one should expect that $a$ is on average $0$ and $b$ is on average $1$ (with some variation around this). I did this numerically and while a distribution of $a$ that is symmetric around $0$ looks credible, the average for $b$ is smaller than $1$. Here is some basic code in R that does this experiment.

n_pop  <- 100

n_sample <- 20

draws <- 1000

M <- matrix(c(1:(2*draws)), nrow=draws)

for (j in 1:draws) {

x <- sort(sample(1:n_pop,n_sample,replace=FALSE))

y <- sort(sample(1:n_pop,n_sample,replace=FALSE))

xy_model <- lm(y~x)

M[j,] <- coef(xy_model)

}

mean(M[,1])

mean(M[,2])

sd(M[,1])

sd(M[,2])

hist(M[,1],breaks=20)

hist(M[,2], breaks=20)


So what is going on? What is the expected value of the coefficient $b$ and what is its distribution?

edit: some numerical evidence that the coefficient $b$ is not $1$ (and $a$ is not zero either), change the number of draws to 10000 to get higher confidence (takes a few seconds to run on my computer) and add the two lines

t.test(M[,2], mu=1, conf.level=0.99)

t.test(M[,1], mu=0, conf.level=0.99)

gives a 99% confidence interval of $[0.9757861, 0.9828772]$ for $b$ and a p-value of $10^{-16}$ for $b=1$. Similarly the $a$ coefficient is around $1$ and almost certainly not zero.

I suspect this is related to the fact that ordinary least squares minimizes the squared distance in the $y$ direction which is not symmetric with respect to $x$ and $y$. A Deming regression which minimizes euclidean distance to the line should get the $a=0$ and $b=1$ that I expected.

• Just to check your code have you tried running it when you set $y_i=x_i$ to see if this gives $b=1$ Oct 17 '17 at 9:14
• If you compute the linear model lm(x~x) you get x = 0 +1*x with zero error as expected. Oct 17 '17 at 9:56
• You say that the average for $b$ is smaller than $1$. Independently from the number of repetitions you do, you will see almost surely an estimated $\hat b\neq 1$. The simplest thing I can think of is to store all the estimated $\hat b$ and compute a $95\%$ confidence interval for that. If $1$ falls inside that interval then there is no statistical evidence for stating that $b$ is different from $1$.
– Ilis
Oct 17 '17 at 12:14
• See here to see how to do this test en.wikipedia.org/wiki/… Oct 17 '17 at 12:42