least square problem normal equations Can you give an example which shows that loss of information can occur in forming
the normal equations. How is accuracy improved using iterative improvement?
Thank you
 A: Forming the normal equations doesn't lose any information. However solving them is more prone to numerical error because the condition number of $A^T A$ can be the square of the condition number of $A$. 
If you look at Section 1.5 of http://www.cs.ubc.ca/~rbridson/courses/542g-fall-2008/notes-oct1.pdf there is a simple example that illustrates the problem. Choose $f=(1,-1)^T$ and try the problem in Matlab/Octave. Computing $A^{-1}f$ produces $(0,-1)^T$ which is good, but solving $(A^T A)^{-1}A^T f$ produces a warning and the result $(3.8320,2.8340)^T$.
The matrix in question is $A=\begin{bmatrix}1+10^{-8} & -1 \\ -1 & 1\end{bmatrix}$. $\kappa(A) \approx 4 \cdot 10^8$, $\kappa(A^TA) \approx 1.0648 \cdot 10^{17}$.
To answer the iterative accuracy improvement question, you need to indicate what iterative method you are using.
A: iirc, if you try to fit a polynomial of degree n at equally spaced points,
you get a Hilbert matrix of size n, which rapidly becomes badly ill-conditioned
as n increases.
