Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & -1 & -1\\ 1 & -1 & -1 \end{matrix}$ and

X'= \begin{matrix}1 & 0 & 0\\ 1 & 0 & 0\\ 1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 & 1\\ 1 & 0 & 1 \end{matrix}

with response vector $Y=[Y_{11} Y_{12} Y_{21} Y_{22} Y_{31} Y_{32}]^{T}$

Do these two matrices give you the same model?

I've had some trouble trying to figure out how, given that $\beta = (\beta_1 \beta_2 \beta_3)^T$, these two designs can give the equivalent model. Don't they have entirely different values of $\beta$ for each $Y_{ij}$?

EDIT: The constraints are $\sum_{i=1}^{3}\beta_i = 0$

share|cite|improve this question
No response vector given? – J. M. May 10 '12 at 15:45
added, sorry about that – Emir May 10 '12 at 15:51
Consider $y=(1,-1,1,-1,1,-1)$. – Chris Taylor May 10 '12 at 16:18

$\beta=X^\dagger Y$ where $X^\dagger=(X^t X)^{-1} X^t$ is the pseudo inverse of X.

If you compute the pseudo inverse of both $X$ and $X'$ you will see that they give different matrix (octave code below). Hence your models will, most likely, be different. Now there might be some $Y$ such that the $\beta$ will be the same (i.e. such that $(X')^\dagger Y = (X)^\dagger Y$) but I'm thinking this is somehow irrelevant here (however, as J.M. points out, this case can happen).

X = [1 1 0; 1 1 0; 1 0 1; 1 0 1; 1 -1 -1; 1 -1 -1];

Xb = [1 0 0; 1 0 0; 1 1 0; 1 1 0; 1 0 1; 1 0 1];



Edit: a counter example shows the same: let Y=[1;2;3;4;5;6], then X\Y yields [3.5;-2;0] and Xb\Y yields [1.5;2;4] where A\b operation amounts to applying the pseudo inverse of A to the vector b.

share|cite|improve this answer
On the other hand, things like the Anscombe quartet make me leery of quickly concluding that two different design matrices cannot yield the same model... – J. M. May 10 '12 at 16:01
yes that is absolutely right! I just think that with respect to the question which was asked in a general sense, the general answer can be "no there is a possibility to get two different models". However, as mentioned, there might be some $Y$ for which the models are indeed the same. – tibL May 10 '12 at 16:11

I suspect that what is intended is for you to check if the models give the same fit to the data after fitting a least-squares model, i.e. that the predicted value of the outcome vector, $\hat Y$ is the same between models. $\hat Y$ turns out to be the projection of $Y$ onto the column space of $X$, so this is equivalent to $X$ and $X'$ having the same columns space.

$X$ and $X'$ do in fact have the same column space and so give the same fit. To get the second column of $X'$ from $X$, so $\frac 1 3 x_1 - \frac 1 3 x_2 + \frac 2 3 x_3$ where $X = [x_1 x_2 x_3]$, and to get the third column do $\frac 1 3 x_1 - \frac 1 3 x_2 - \frac 1 3 x_3$.

Alternatively, we can check that the projection matricies $P(X) = X (X^TX)^{-1} X^T$ of $X$ and $X'$, and they do: both are $6 \times 6$ block diagonal with diagonal matricies all equal to $$D = \begin{pmatrix} \frac 1 2 & \frac 1 2 \\ \frac 1 2 & \frac 1 2 \end{pmatrix}.$$

An alternative way of phrasing this equivalency between models is that if the "true" model $M_0$ can be expressed as $Y = X\beta + \epsilon$ then it can also be expressed as $Y = X'\beta' + \epsilon$ for a different parameter $\beta'$ and so the models are equivalent.

EDIT: With the added constraint, they no longer result in the same model, see comment below. I think the point is that even though without the constraints these models are equivalent in some sense, adding constraints is enough to turn them into different models.

share|cite|improve this answer
I forgot to mention that the $\beta_i$ must sum to zero. Does this constraint change your response? – Emir May 10 '12 at 17:21
Yes, it matters a lot. I think this should take into account the constraint you gave, but I'm not 100% sure. Write $\beta = (\beta_1, \beta_2, 1 - \beta_1 - \beta_2)$, and write $\beta_\star = (\beta_1, \beta_2)$. Find $L$ such that $L \beta_\star = \beta$. Then form design matricies $X_\star = X L$ and $X'_\star = X'L$ and check if they have the same column space (I don't see why they should just because $X$ and $X'$ do). – guy May 10 '12 at 18:37
In previous comment, obviously meant to put $\beta = (\beta_1, \beta_2, -\beta_1 - \beta_2)$. – guy May 10 '12 at 18:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.