The Eigen library’s documentation on how to solve linear least squares shows three examples: https://eigen.tuxfamily.org/dox/group__LeastSquares.html
Using SVD decomposition, using QR decomposition and using normal equations.
If I have an overconstrained problem $Ax = y$ (more data points y than variables x) then I can transform it into a problem that isn’t overconstrained by multiplying the transpose of A to the left of both sides of the equation $A^TAx = A^Ty$ this problem has the same amount of datapoints $y$ as variables to fit and I can calculate an exact solution for $x$.
Is this solution x really also the one that minimizes the squared error of the original problem? How is this possible? Both the matrix $A^TA$ as well as the vector $A^Ty$ are smaller than the ones of the original problem and thus contain less information.
Or is the Eigen documentation misleading and this doesn’t actually yield an optimal solution in the least squares sense (numerical stability aside)?