# Least squares method to a square system of equations that has a unique solution

I am new to the least square and understand that Least square is applied to the overdetermined system to get the best fit but what happens if we apply the least squares method to a square system of equations that has a unique solution. For linear fit, all points should be in the same line if the solution is unique. IS there any general explanation of the above question? Also, why will it happen in the general case?

• If the system of equations is square and has a unique solution (that means invertible) then we have $A^T A x = A^T b$ and hence multiplying by $(A^T)^{-1}$ gives $Ax=b$ or $x = A^{-1} b$. – copper.hat Nov 21 '17 at 4:54

Using the notation in the corresponding Wikipedia article, the normal equations to find $\beta$ minimizing $\Vert y - X\beta \Vert_2$ ($X$ is an $m\times n$ matrix with $m \geq n$) are $X^\intercal X \beta = X^\intercal y$.

The solution of this system is $\beta = X^+ y$ where $X^+ = (X^\intercal X)^{-1} X^\intercal$ is the Moore-Penrose pseudoinverse of $X$ (assuming $X$ has linearly independent columns so that $X^\intercal X$ is nonsingular).

As per your question, when $X$ is a square invertible matrix, the Moore-Penrose pseudoinverse is exactly the ordinary inverse since $$X^+ = (X^\intercal X)^{-1} X^\intercal = X^{-1} X^{-\intercal} X^{\intercal} = X^{-1}.$$ In this case, $\beta = X^{-1} y$.

Suppose we are given vectors $y,w$ and a matrix $X$, not necessarily invertible. Then to find the $w$ that minimizes $\dfrac{1}{2} \Vert y - Xw\Vert_2^2$

First, let $f(w) = \dfrac{1}{2} \Vert y - Xw\Vert_2^2 = \dfrac{1}{2}(y - Xw)^T (y - Xw)$

Then taking the gradient yields: $\nabla f(w) = X^T(y - Xw^*) = 0 \implies w^* = X^{\dagger}y$

By first-order optimiality condition (something well known in nonlinear optimization), $w^*$ is the minimizer of the convex objective function $f(w)$.

($X^{\dagger} = (X^TX)^{-1}X^T$ is the pseudo-inverse of $X$)