I am working with simulation software to generate training data. Specifically, I want to create a regression model predicting the weight of plastic parts from an injection molding process.

Besides an accurate prediction, I am interested in quantifying a prediction interval, which I am computing as:

$$\hat\theta x_{new}\pm t_{\frac{\alpha}{2},n-p}s\sqrt{1+x_{new}(X^\top X)^{-1}x_{new}^\top}$$

Since I am obtaining the training data from simulation, I would like to specify the design matrix $X$ in such a way, that I tend to minimize the average prediction interval (in a relevant interval for future predictions $[L,U]$). In the simple two-dimensional case, I can derive the following statements: Since $$SE(y)=MSE*\left[ 1+\frac{1}{n}+\frac{(x_{new}-\bar X)^2}{\sum_{\forall i} (x_i-\bar X)^2} \right]$$

it makes sense to center the data collection efforts on the interval where future predictions are to be made to minimize $x_{new}-\bar X$, , while the observations $x_i$ should spread as widely from the center as possible, to maximize the term $\sum_{\forall i} (x_i-\bar X)^2$. Intuitively, I understand that a straight line fit is 'fixed' more by distant points than by close points.

I would like to generalize this to the multivariate regression setting, to specify how to simulate data (given the objective of minimizing the uncertainty intervals). From this post:

Multicollinearity: Why does highly correlated columns in the design matrix lead to high variance of the regression coefficient?

I understand that it would make sense to generate the data s.t. the columns of $X$ are not linearly dependent. Intuitively, this sounds to me like a similar version to the above statement. I am left with the question: What is a mathematically sound approach of specifying the Design matrix to minimize the average prediction interval in the multivariate regression case?


1 Answer 1


You always can generate a "degenerate case" where the number of $\beta$s equals the number of observations (without complete colineariaty). In this case you will have a $0$ variance, i.e., $\hat{Y} = Y$.

For a more realistic case, when you have $n>>p$ and you cannot control the noise term $\varepsilon$, then $\mathrm{X}$ with (piarwise) orthogonal columns will give you the most "stable" $\mathrm{(X'X)}^{-1}$ matrix.

  • $\begingroup$ Thank you very much for the input! Am I right in the assumption that I should thus try to find columns s.t. the dot products of pairwise column vectors get as close to zero as possible? $\endgroup$
    – Sean
    Jul 19, 2017 at 19:00
  • $\begingroup$ Yes, it is right. $\endgroup$
    – V. Vancak
    Jul 19, 2017 at 19:01
  • $\begingroup$ Again, thanks alot! It seems to me that this is a nonlinear optimization problem, where the objective is to minimize $\sum_{\forall i,j; i\not=j}|v_i \cdot v_j|$ with the variables $x_i,x_j$ representing the vector entries for vector $i$ and $j$ respectively. Do you see any heuristical approach to intelligently, not necessarily optimally, selecting the vectors, i.e. the $x_is$ & $x_js$? $\endgroup$
    – Sean
    Jul 19, 2017 at 20:08
  • $\begingroup$ Vectors like $(x_1, x_2, x_3, 0,0,0)$ and $(0,0,0,y_1, y_2, y_3)$ are trivial examples. Maybe following this logic will help. Anyway, you're welcome! (BTW, you can up-vote/accept the answer if it helps you. $\endgroup$
    – V. Vancak
    Jul 20, 2017 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.