Linear vs Orthogonal polynomial for fitting

Question

I am neither a mathematician nor do I possess a large knowledge in maths so this could also be completely false. But I read that if you use an orthogonal function like the Chebychev-polynomial to construct the background of a set of datapoints, in my case a diffraction experiment, you will not end up in having your polynomial function starting to oscillate once you approach a higher order. I think you can see this effect whenever you try to fit something using simple polynomial functions of higher order like Excel or Origin does it. Once you approach a higher order of your function it doesn't become better but starts to oscillate between the datapoints. So your solution becomes worse at some point. I was doing a Rietveld-refinement when this happened to me. So, after reading through a few books I switched to a Chebychev-function which gave a much better solution. I tested the function by refining even more parameters and it just stopped at some point. It didn't become better or worse. So there was no oscillation happening here. For my specific case this seems to have a physical reason, being that the background parameters themselves and the background parameters plus the reflexes are highly correlated.

But appart from any physical interpretation is there a simple mathematical explanation to why the Chebychev-function is so much better at describing a function even though you over-parameterize it?

Answer 1

Your polynomial fit solves a linear least squares problem. Don't be fooled by the name: "linear" means that the polynomial values are linear in the polynomial coefficients, not that the polynomial is a line/first-order.

The type of problem that is solved here is "find a polynomial that is the best possible approximation to these specific $y$-values at these specific $x$-values". If you let the polynomial degree equal the number of points you have, the solution is the Lagrange interpolant.

Using orthogonal polynomials, you don't solve a least squares problem (although you could). No, the approximation here is a projection, a geometric operation, albeit one in more dimensions than we are used to. It involves calculating an inner product, an integral, like this: let $f_i(x)$ be the polynomials and $g(x)$ your data: $$g(x)\approx \sum_i f_i(x)\int f_i(t) g(t)dt$$ If your data is discrete, this integral must be approximated by, say, trapezoidal integration, which is essentially assuming $g(x)$ is piecewise linear. So now, you're solving the very different problem "find the best polynomial approximation to this piecewise linear function", and a highly oscillatory polynomial, although it may be a good fit at specific points, is not globally a good solution.