From what I understand, when doing a least squares regression with a Vandermonde matrix, you're essentially solving the equation


Where $y$ is a vector of $y$-values, $X$ is the Vandermonde matrix, and $a$ is a vector of coefficients.

When you solve this equation for $a$,


You get the above expression for $a$.

My understanding is that this should be a solution to the set of equations. However, it is possible to fit $n$ points of data to a $k$-th degree polynomial, where $n>k$. This would imply that there are more equations than unknowns, which results in no possible solutions. However, the vector a can be calculated. This resulting vector does not completely perfectly satisfy


But instead is a good approximation, as with regression.

Why can we get a value for $a$? As I do not believe this could be possible if we were dealing with $n$ equations, and $k$ unknowns. Where does the matrix solution differ from solving $k$ unknowns with $n$ equations?


First, a little explanation of what is happening when you solve for $a$ in

$\vec{a}=(X^TX)^{-1}X^T\vec{y}$ .

The $(X^TX)^{-1}X^T$ part is the Moore-Penrose pseudoinverse. It uses $X$'s Gramian matrix (the $X^TX$), which is a square positive semidefinite matrix yielding the same eigenvectors as $X$ while squaring its eigenvalues. Taking the inverse of this Gramian matrix and multiplying it by $X^T$ actually performs a least square fit (see this and this reference). It is a property of the pseudoinverse.

As for your question about dealing with $n$ equations and $k$ unknowns, there are two possible cases (I suppose the rank of your matrix is at least equal to $k$):

  • The number of (linearly independent) equations is greater than the number of unknowns ( $n > k$, matrix is overdetermined). In this case, either a single exact solution or none exist. The least squares regression is commonly used in this case. It will give the coefficients $\vec{a}$ that minimizes the euclidean norm of the errors: $\min \epsilon \: \left( \hat{y} = X\vec{a} + \epsilon \right) $.
  • The number of (linearly independent) equations is equal or less than the number of unknowns ( $n \le k$, matrix is full rank or underdetermined). In this case, either a single exact solution (when full rank) or many (up to an infinity) of them exist (when full rank or underdetermined). The pseudoinverse will construct the solution that minimizes the euclidean norm among all solutions in the matrix the nullspace: $\min \hat{a} \: \left( \hat{y} = X\vec{a} \right) $.

The equation that you have provided:

$$ a=(X^TX)^{-1}X^Ty $$

as you stated performs least squares regression, more specifically Ordinary Least Squares. The least squares approach provides a solution to an overdetermined system (more linearly independent equations than unknowns) by providing coefficients ($\hat{a}$) that minimize the sum of the squares of the differences between the y-values in the given dataset and those predicted by the linear function.

In simpler terms, if you acquired the coefficients using the least squares solution

$$ \hat{a}=(X^TX)^{-1}X^Ty $$

and substituted this coefficient vector back into your system of equations


you would generate predicted (or fitted) y-values $\hat{y}$ that are your original y-values projected onto the new line of best fit. The difference between your original y-values and these predicted y-values are your residuals

$$ \hat\varepsilon=y-\hat{y} $$

which describe the error between the true y-values and the fitted y-values. The goal of regression is determine a system that best matches your data- and one way this is accomplished is by essentially minimizing the error between your model prediction and your actual data. One way to do this is by squaring these errors and summing them $$ \sum_{i=1}^{n}(y-\hat{y})^2 $$

This provides an indicator of how well the predicted coefficents fit the data, more specifically by minimizing this indicator you are left with the Best Linear Unbiased Estimate.

Note: There are several generalizations, over simplifications and assumptions not mentioned in this answer. It was was written to provide clarification and a simplified explanation, not complete accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.