Why does feature scaling by a constant vs. by a function produce different results in linear regression?

If I have some data points $$(x_1, x_2,\dots,x_n,y)$$ where $$y$$ is the dependent variable and want to perform linear regression. If I perform regression with $$x_1, x_2,\dots,x_n$$ and $$y$$ vs. scaling one or multiple $$x_i$$ feature by some constant (say set $$\hat x_i = x_i/c$$), the sum of square errors of the two curves I get are identical, and the predicted $$y$$ value for some input are also the same for both (provided for some $$x_i$$ in the first model I divide that by $$c$$ before using it in the second model).

This isn't true if I apply a monotonic function (say logarithmic, $$\hat x_i = log x_i$$))? Why is this the case? (assuming for this case all the x's are positive).

• Consider the case where the error in the original linear regression is zero.
– user856
Commented Jul 14, 2019 at 19:38

Linear regression assumes that the dependent variable is a linear function of independent variables. Consider $$x = (x_1, x_2, ...x_n)$$ as input variables and $$y$$ as the target variable. LR finds the weights $$(w_0, w_1,..w_n)$$ in such a way to reduce the mean square error $$MSE = \sum_{i=1}^{N} {(y_i-\hat{y_i})}^2$$ where $$$$\hat{y_i} = f(x_i) = w_0 + w_1x_1+w_2x_2+..+w_nx_n$$$$ When each of the features are scaled by some value say $$c_i$$, then $$$$\begin{split} \hat{y_i} = f(x_i) &= w_0 + w_1x_1+w_2x_2+..+w_nx_n \\ &= w_0 + \frac{w_1}{c_1} (c_1x_1) + ..+ \frac{w_n}{c_n} (c_nx_n) \end{split}$$$$
i.e. when the respective features are scaled by $$c_i$$, LR scales the weights so as to get the same target variable. Consider the case where logarithm is applied to each feature, then
$$$$\begin{split} \hat{y_i} = f(x_i) &= a_0 + a_1\log(x_1)+a_2\log(x_2)+..+a_n\log(x_n) \\ &= a_0 + log(x_1^{a_1})+..+log(x_n^{a_n}) \\ &= a_0 + log \prod_{i=1}^{n} x_i^{a_i} \hspace{2ex}\text{(non-linear)}\\ \end{split}$$$$
i.e. when you apply logarithm, each feature is compressed/scaled, which depends on the value being scaled i.e. $$x_i$$. This scale(which is not constant) value can't be captured by the LR. When the new input $$k = (k_1, k_2,..k_n)$$ comes at the inference time, and you apply non-linear function like logarithm, the features will be scaled differently again, than from the dataset. The bottomline is that logarithm introduces non-linearity into the features and LR can't capture this information.