# Cost Function Confusion for Ordinary Least Squares estimation in Linear Regression

Just wanted to check that my current understanding of linear regression is correct and address some confusion I have with the cost function used in OLS estimation. My current understanding is this:

Given a data set:

$$\{y_i, x_{i1}, x_{i2}, ... ,x_{ip}\}_{i=1}^{N}$$

The Multiple Linear Regression Model that most accurately describes the relationship between dependent variable $$y$$ and independent variables $$x_1, x_2, ... , x_p$$ is the linear function:

$$y = \beta_0x_0 + \beta_1x_1 + ... + \beta_px_p = \sum_{j=0}^p\beta_j(x_j)$$

such that $$\forall y_i$$ (where $$y_i = \beta_0x_{i0} + \beta_1x_{i1} + ... + \beta_px_{ip} + \epsilon_i$$)

the sum of squared residuals: $$\sum_{i}^{N}\epsilon^2 = \sum_{i}^{N}(y_i - (\beta_0x_{i0} + ... + \beta_px_{ip}))^2$$ is minimized.

This all comes from the following sources:

https://en.wikipedia.org/wiki/Ordinary_least_squares#Matrix/vector_formulation

https://en.wikipedia.org/wiki/Linear_regression#Simple_and_multiple_linear_regression

https://en.wikipedia.org/wiki/Linear_least_squares

My confusion comes from other sources I have looked at:

https://stackoverflow.com/questions/34148912/feature-scaling-normalization-in-multiple-regression-analysis-with-normal-equa

https://machinelearningmedium.com/2017/08/11/cost-function-of-linear-regression/

which say that the Multiple Linear Regression Model that most accurately describes the relationship between $$y$$ and $$x_1, x_2, ... x_p$$ is the same linear function I originally defined above but that the coefficients $$\beta_0, \beta_1, ..., \beta_p$$ for it are those which minimize the cost function:

$$\frac{1}{2N}\sum_{i=1}^{N}(y_i - (\beta_0x_{i0}+ \beta_1x_{i1} + ... + \beta_px_{ip}))^2$$

So my question is this: Is my current understanding correct? and which of these cost functions should I be using?

Either cost function is fine. Since they are just constant multiples of each other, minimising one is equivalent to minimising the other (will result in same fitted $$\beta$$ coefficients).
As already mentioned - all of these functions are equivalent to each other up to a multiplication by some constant. Moreover, you can view the sum of squares as slightly more algebraic approach as $$\sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i)^2 = \|\mathrm{y} - X\beta\|^2,$$ which is equivalent to minimizing the Euclidean norm itself. That is, finding an orthogonal projection. While,
$$\frac{1}{N}\sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i)^2 = \frac{1}{N}\sum_{i=1}^n (y_i - g(x_i))^2,$$ where $$g(x_i)$$ is the mean of $$y_i$$. This approach may be referred as minimizing the empirical mean squared error (MSE) which is a more statistical POV.
• I can understand how $\sum_{i=1}^{n}(y_i-\beta_0-\beta_1x_i)^2 = \Vert y - X\beta \Vert ^2$ but what I do not understand is the geometric interpretation: "finding an orthogonal projection". What is this projection orthogonal to exactly? Is it something like this: textbooks.math.gatech.edu/ila/projections.html ? Commented May 15, 2020 at 18:47
• @bmanicus131 See here for example: people.eecs.ku.edu/~jhuan/EECS940_S12/slides/… The projected $y$ is orthogonal to the errors $y - \hat{y}$. Namely, $y$ is projected onto the column space of $X$, where $\hat{\beta}$s are the coordinates of $\hat{y}$ in $C(X)$. Commented May 15, 2020 at 20:11