# design matrices

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & -1 & -1\\ 1 & -1 & -1 \end{matrix}$ and

X'= \begin{matrix}1 & 0 & 0\\ 1 & 0 & 0\\ 1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 & 1\\ 1 & 0 & 1 \end{matrix}

with response vector $Y=[Y_{11} Y_{12} Y_{21} Y_{22} Y_{31} Y_{32}]^{T}$

Do these two matrices give you the same model?

I've had some trouble trying to figure out how, given that $\beta = (\beta_1 \beta_2 \beta_3)^T$, these two designs can give the equivalent model. Don't they have entirely different values of $\beta$ for each $Y_{ij}$?

EDIT: The constraints are $\sum_{i=1}^{3}\beta_i = 0$

-
No response vector given? – J. M. May 10 '12 at 15:45
Consider $y=(1,-1,1,-1,1,-1)$. – Chris Taylor May 10 '12 at 16:18

$\beta=X^\dagger Y$ where $X^\dagger=(X^t X)^{-1} X^t$ is the pseudo inverse of X.

If you compute the pseudo inverse of both $X$ and $X'$ you will see that they give different matrix (octave code below). Hence your models will, most likely, be different. Now there might be some $Y$ such that the $\beta$ will be the same (i.e. such that $(X')^\dagger Y = (X)^\dagger Y$) but I'm thinking this is somehow irrelevant here (however, as J.M. points out, this case can happen).

X = [1 1 0; 1 1 0; 1 0 1; 1 0 1; 1 -1 -1; 1 -1 -1];

Xb = [1 0 0; 1 0 0; 1 1 0; 1 1 0; 1 0 1; 1 0 1];

inv(X'*X)*X'

inv(Xb'*Xb)*Xb'

Edit: a counter example shows the same: let Y=[1;2;3;4;5;6], then X\Y yields [3.5;-2;0] and Xb\Y yields [1.5;2;4] where A\b operation amounts to applying the pseudo inverse of A to the vector b.

-
On the other hand, things like the Anscombe quartet make me leery of quickly concluding that two different design matrices cannot yield the same model... – J. M. May 10 '12 at 16:01
yes that is absolutely right! I just think that with respect to the question which was asked in a general sense, the general answer can be "no there is a possibility to get two different models". However, as mentioned, there might be some $Y$ for which the models are indeed the same. – tibL May 10 '12 at 16:11

I suspect that what is intended is for you to check if the models give the same fit to the data after fitting a least-squares model, i.e. that the predicted value of the outcome vector, $\hat Y$ is the same between models. $\hat Y$ turns out to be the projection of $Y$ onto the column space of $X$, so this is equivalent to $X$ and $X'$ having the same columns space.

$X$ and $X'$ do in fact have the same column space and so give the same fit. To get the second column of $X'$ from $X$, so $\frac 1 3 x_1 - \frac 1 3 x_2 + \frac 2 3 x_3$ where $X = [x_1 x_2 x_3]$, and to get the third column do $\frac 1 3 x_1 - \frac 1 3 x_2 - \frac 1 3 x_3$.

Alternatively, we can check that the projection matricies $P(X) = X (X^TX)^{-1} X^T$ of $X$ and $X'$, and they do: both are $6 \times 6$ block diagonal with diagonal matricies all equal to $$D = \begin{pmatrix} \frac 1 2 & \frac 1 2 \\ \frac 1 2 & \frac 1 2 \end{pmatrix}.$$

An alternative way of phrasing this equivalency between models is that if the "true" model $M_0$ can be expressed as $Y = X\beta + \epsilon$ then it can also be expressed as $Y = X'\beta' + \epsilon$ for a different parameter $\beta'$ and so the models are equivalent.

EDIT: With the added constraint, they no longer result in the same model, see comment below. I think the point is that even though without the constraints these models are equivalent in some sense, adding constraints is enough to turn them into different models.

-
I forgot to mention that the $\beta_i$ must sum to zero. Does this constraint change your response? – Emir May 10 '12 at 17:21
Yes, it matters a lot. I think this should take into account the constraint you gave, but I'm not 100% sure. Write $\beta = (\beta_1, \beta_2, 1 - \beta_1 - \beta_2)$, and write $\beta_\star = (\beta_1, \beta_2)$. Find $L$ such that $L \beta_\star = \beta$. Then form design matricies $X_\star = X L$ and $X'_\star = X'L$ and check if they have the same column space (I don't see why they should just because $X$ and $X'$ do). – guy May 10 '12 at 18:37
In previous comment, obviously meant to put $\beta = (\beta_1, \beta_2, -\beta_1 - \beta_2)$. – guy May 10 '12 at 18:45