# Polynomial Regression and Multicollinearity

I am trying to do a least square fit of the following model (by finding the $\beta$) but it turns out the right-hand side X matrix has a condition number of about 3000. I wonder if there is a way to reduce the condition number.

$$\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}= \begin{bmatrix} x_1^3 - x_0^3 & x_1^2 - x_0^2 & x_1 - x_0 \\ x_2^3 - x_0^3 & x_2^2 - x_0^2 & x_2 - x_0 \\ \vdots & \vdots & \vdots\\ x_n^3 - x_n^3 & x_n^2 - x_0^2 & x_n - x_0 \end{bmatrix} \begin{bmatrix} \beta_1 \\ \beta_2 \\ \beta_3 \end{bmatrix} + \text{Error}$$

Note that:

1. $x_1, x_2,\ldots,x_n$ are not equal distance
2. I am forcing the fitted function to be $0$ at $x_0$

I wonder if there is any way to make the $X$ matrix less linearly dependent.

Thanks.

You could apply the Gram-Schmidt process to the column space and get three vectors orthogonal to each other and spanning the same column space. That's the same as finding a $3\times3$ matrix $M$ such that $XM$ has orthogonal columns. Then instead of $Y=X\beta+\text{error}$ you'd have $Y=(XM)(M^{-1}\beta)+\text{error}$. You would then find least-squares estimates of the three entries in the $3\times1$ column vector $\gamma=M^{-1}\beta$. Finally, you could use $\beta=M\gamma$ to get least-squares estimates of the three entries $\beta_1,\beta_2,\beta_3$.
(Given the order in which you've written the columns, I might have called the three parameters to be estimated $\beta_3,\beta_2,\beta_1$, in that order.)
To be a bit more explicit about $M$: I don't know about efficiency of algorithms in linear algebra, but if I were using R to do this problem, I might do something like this:
• Regress the second column of $X$ on the third, forcing the intercept to be $0$, so you get $X_{[\, ,2]} = p X_{[\,,3]}+\text{residuals}$. Put the residuals in place of the second column. Now the column space of this new $X$ is the same as that of the old $X$ but the second and third columns are orthogonal.
• Regress the first column of $X$ on the second and third columns of the (original) $X$, forcing the intercept to be $0$, so you get $X_{[\,,1]} = q X_{[\,,2]}+r X_{[\,,3]}+\text{residuals}$. Put the residuals in place of the first column. Now the new first column is orthogonal to the second and third columns (regardless of whether you use the new second column or the original one) and the three columns are now mutually orthogonal.
• Then we have $$\text{new }X = \left(\text{old }X\right)\begin{bmatrix} 1 & 0 & 0 \\ -q & 1 & 0 \\ -r & -p & 1 \end{bmatrix}.$$ There you have the matrix $M$.
• To orthogonalize when constructing $X$ would give the same matrix that I called $XM$, I would think. – Michael Hardy Aug 24 '13 at 23:30