1
$\begingroup$

I am trying to do a least square fit of the following model (by finding the $\beta$) but it turns out the right-hand side X matrix has a condition number of about 3000. I wonder if there is a way to reduce the condition number.

$$ \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}= \begin{bmatrix} x_1^3 - x_0^3 & x_1^2 - x_0^2 & x_1 - x_0 \\ x_2^3 - x_0^3 & x_2^2 - x_0^2 & x_2 - x_0 \\ \vdots & \vdots & \vdots\\ x_n^3 - x_n^3 & x_n^2 - x_0^2 & x_n - x_0 \end{bmatrix} \begin{bmatrix} \beta_1 \\ \beta_2 \\ \beta_3 \end{bmatrix} + \text{Error} $$

Note that:

  1. $x_1, x_2,\ldots,x_n$ are not equal distance
  2. I am forcing the fitted function to be $0$ at $x_0$

I wonder if there is any way to make the $X$ matrix less linearly dependent.

Thanks.

$\endgroup$
1
$\begingroup$

You could apply the Gram-Schmidt process to the column space and get three vectors orthogonal to each other and spanning the same column space. That's the same as finding a $3\times3$ matrix $M$ such that $XM$ has orthogonal columns. Then instead of $Y=X\beta+\text{error}$ you'd have $Y=(XM)(M^{-1}\beta)+\text{error}$. You would then find least-squares estimates of the three entries in the $3\times1$ column vector $\gamma=M^{-1}\beta$. Finally, you could use $\beta=M\gamma$ to get least-squares estimates of the three entries $\beta_1,\beta_2,\beta_3$.

(Given the order in which you've written the columns, I might have called the three parameters to be estimated $\beta_3,\beta_2,\beta_1$, in that order.)

To be a bit more explicit about $M$: I don't know about efficiency of algorithms in linear algebra, but if I were using R to do this problem, I might do something like this:

  • Regress the second column of $X$ on the third, forcing the intercept to be $0$, so you get $X_{[\, ,2]} = p X_{[\,,3]}+\text{residuals}$. Put the residuals in place of the second column. Now the column space of this new $X$ is the same as that of the old $X$ but the second and third columns are orthogonal.
  • Regress the first column of $X$ on the second and third columns of the (original) $X$, forcing the intercept to be $0$, so you get $X_{[\,,1]} = q X_{[\,,2]}+r X_{[\,,3]}+\text{residuals}$. Put the residuals in place of the first column. Now the new first column is orthogonal to the second and third columns (regardless of whether you use the new second column or the original one) and the three columns are now mutually orthogonal.
  • Then we have $$ \text{new }X = \left(\text{old }X\right)\begin{bmatrix} 1 & 0 & 0 \\ -q & 1 & 0 \\ -r & -p & 1 \end{bmatrix}. $$ There you have the matrix $M$.
$\endgroup$
  • $\begingroup$ I wonder if it is too late if we explicitly orthogonalize X. Orthogonalization is what most regression packages do internally when they solve the normal equation via pinv or QR. I think explicit orthogonalization will appear to reduce the condition number but won't improve the numerical stability. Maybe we should orthogonalize when constructing X. $\endgroup$ – Tom Bennett Aug 24 '13 at 22:01
  • $\begingroup$ To orthogonalize when constructing $X$ would give the same matrix that I called $XM$, I would think. $\endgroup$ – Michael Hardy Aug 24 '13 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.