Does removing the intercept from linear regression reduce the bias from the predictor MSE? This is a deceptively simple question but one that, to the best of my knowledge, has no good proof/answer online. If anyone can point to a source please do, because I failed to find any.
The question is: if we remove the intercept from a linear regression model, does it reduce the bias component of the MSE (Mean squared error)?
 A: Suppose we have 2 simple linear regression models $M_1$ and $M_2$, each described by the following model equations
$$M_1: Y = \beta_1 X + \beta_0 + \epsilon $$
$$M_2: Y = \beta_2 X + \epsilon $$
In plain English, $M_1$ is a linear regression with the intercept, and $M_2$ is a linear regression without the intercept for some set of data $\{X_1, X_2, \dots X_n\}$ and $\{Y_1, Y_2, \dots Y_n\}$.
We can write the predictors of $Y$ for the 2 models as
$$\hat{Y}_{1,i} = \hat{\beta_1} X_i + \hat{\beta_0}$$
$$\hat{Y}_{2,i} = \hat{\beta_2} X_i $$
Note that in the general case absolutely nothing implies that $\hat{\beta_1} = \hat{\beta_2}$ or that they are even "close". Please check this answer to a related question on stack overflow which provides a nice visual depiction
As per the definition of Mean Squared Error (MSE) and Bias for predictors and the expression for $\hat{\beta_0} = \bar{Y} - \bar{X}\hat{\beta_1}$, we can write
$$ Bias(\hat{Y}_{1,i}) = (\frac{1}{n}\sum_{i=1}^{n}(\hat{Y}_{1,i} - Y_i))^2 = (\frac{1}{n}\sum_{i=1}^{n}(\hat{\beta_1} X_i + \hat{\beta_0} - Y_i))^2 = (\frac{1}{n}\sum_{i=1}^{n}(\hat{\beta_1} X_i + \bar{Y} - \bar{X}\hat{\beta_1} - Y_i))^2  =  (\hat{\beta_1}\frac{1}{n}\sum_{i=1}^{n}(X_i - \bar{X}) + \frac{1}{n}\sum_{i=1}^{n}(\bar{Y} - Y_i))^2 = 0 $$
This is consistent with the usual definition of linear regression (with intercept) being "an unbiased estimator".
We also have
$$ Bias(\hat{Y}_{2,i}) = (\frac{1}{n}\sum_{i=1}^{n}(\hat{Y}_{2,i} - Y_i))^2 = (\frac{1}{n}\sum_{i=1}^{n}(\hat{\beta_2} X_i - Y_i))^2 \geq 0 $$
So unless we are in a degenerate case i.e. all $X_i = Y_i$ we can conclude that $Bias(\hat{Y}_{2,i}) > Bias(\hat{Y}_{1,i}) = 0$
Hence, removing the intercept from a linear regression model will in general increase the bias term in the MSE. We have not proved anything for the full MSE though: we would need to analyse the variance term.
