When does the system of linear equations describing a quadratic regression have infinite solutions I am aware that, if $ax^2+bx+c$ fits $(x_1,y_1),(x_2,y_2),(x_3,y_3),\cdots,(x_n,y_n)$ "best" (i.e. minimises the sum of errors squared $\sum_i\epsilon_i^2$), then we can find $a,b,c$ using the following system of linear equations (shown here in matrix form):
$$\left(\begin{matrix}\sum_i x_i^4 & \sum_i x_i^3 & \sum_i x_i^2 \\ \sum_i x_i^3 & \sum_i x_i^2 & \sum_i x_i \\ \sum_i x_i^2 & \sum_i x_i & \sum_i 1\end{matrix}\right)\left(\begin{matrix}a \\ b \\ c\end{matrix}\right)=\left(\begin{matrix}\sum_ix^2_iy_i \\ \sum_ix_iy_i \\ \sum_iy_i\end{matrix}\right)$$
Where $i$ ranges from $1$ to $n$.
As I understand, at the minimum possible value of $\sum_i\epsilon_i^2$, the partial derivative of $\sum_i\epsilon_i^2$ wrt $a$, $b$, $c$ equals $0$. There always exists a minimum as $\sum_i\epsilon_i^2\geq0$. So we can take the partial derivative of $\sum_i\epsilon_i^2$ and find the critical points. If there is $1$ critical point then that must be the minimum.
But what if there are infinite solutions to the system? When would that happen, and what does that imply?
 A: +I'm not going to concern myself with the "quadratic recession". You are asking about the equation $\begin{bmatrix}a & b & c\\ d & e & f \\ h & i & j\end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}= \begin{bmatrix}y_1 \\ y_2 \\ y_3 \end{bmatrix}$.
That will have a unique solution if and only if the determinant of the matrix is non-zero.   If the determinant is 0, then the matrix maps $R^3$ to a subspace of $R^3$.  If the right side, $\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix}$, is not in that subspace there is no solution, if it is, there are infinitely many solutions.
A way of determining that, is "row reducing the augmented matrix".  Here the augmented matrix is $\begin{bmatrix}a & b & c & y_1\\ d & e & f & y_2\\ h & i & j & y_3\end{bmatrix}$.  if, after row redution, there are no rows where the first three columns are all "0", there is a unique solution.  If there is one or more rows where the first three columns are all "0" but the fourth column is not "0" there is no solution.  If there are one or more rows where the first four columns are "0" and, in each of those,  the fourth column is also "0" then there are infinitely many solutions.
A: Assume that your model is $y = b_0 + b_1x + b_2x^2$, thus given data points $(y_i, x_i)_{i=1}^n$, you have $n$ equations of the following form
$$
y_i = b_0 + b_1 x_i + b_2 x_i^2.
$$
Now, assume that $n=1$, i.e., you have only one point in $\mathbb{R}^2$ that you want to fit a parabola to it, so you have infinitely many such parabolas, i.e., infinitely many solutions of vectors $(b_0, b_1, b_2)$ with $2$ degrees of freedom. If you have $n=2$ distinct data points, then you also have infinitely many solutions of vectors $(b_0, b_1, b_2)$ with $1$ degree of freedom. If you have $n=3$ distinct points then you have a perfect fit, i.e., a unique solution of $(b_0, b_1, b_2)$ for the set of three linear equations $\{y_i = b_0 + b_1 x_i + b_2 x_i^2\}_{i=1}^3$. For every $n > 3$ data points you cannot fit a parabola that passes through all points, thus you'll have the best approximation (in the sense of $l_2$ norm) as the solution as you have an overdetermined system of equations. This fitted parabola may not pass through even one of the data points as it minimizes the squared distance between all available points and not really solves $n$ linear equations.
