3
$\begingroup$

Suppose you have a matrix $X \in \mathbb{R}^{n \times p}$ and the projection matrix in least squares is defined as:

$$ H = X(X^TX)^{-1}X^T $$

where note that the first column of $X$ are ones.

Now suppose that you define $Y = X - \boldsymbol{1}_{n\times 1} C^T$ where $C \in \mathbb{R}^{p \times 1}$ and $C_1 = 0$. So $Y$ is defined as $X$ shifted by some constant amount, $C_i$ subtracted from the $i$-th column of $X.$

How do we show that

$$ H' = Y(Y^TY)^{-1} Y^T = H $$

Here is my attempt (i'm going to drop the bold on the column vector of 1s): $$ H' = Y(Y^TY)^{-1} Y^T \\ = (X - 1C^T)((X - 1C^T)^T(X - 1C^T))^{-1}(X - 1C^T)^T \\ = (X - 1C^T)((X^T - C1^T)(X - 1C^T))^{-1}(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)^{-1}(X^T - C1^T) \\ $$

I stopped here because I know this is going to get messy and will probably involve block matrices, and I think there should be an easier solution than what I'm undertaking?

I believe there's an argument that can be made about how $H$ is spanned by the same basis before and after shifting since $X$ is defined to have a column of ones, so any column of $Y$ can be written as the corresponding column of $X$ and the first column of $X$.

$\endgroup$
1
  • $\begingroup$ This will come down to the fact that the column spaces of $H$ and $H'$ are identical. $\endgroup$ May 5 at 3:43
1
$\begingroup$

The matrix $X$ in this scenario is typically tall and skinny, having many more rows than columns. The matrix $X^TX$ is invertible precisely if the columns of $X$ are linearly independent.

If $u$ is in the column space of $X$ then $u=Xb$ for some column vector $b$ with only as many entries as $X$ has columns, and then a simple algebraic computation shows that $Hu=u.$

But if $u$ is orthogonal to the column space of $X$ then $X^Tu= 0$ and so $Hu=0.$

Thus for any column vector $u$ as "tall" as $X,$ the vector $Hu$ is the orthogonal projection of $u$ onto the column space of $X.$

Therefore, the matrix $H$ depends on $X$ only through the column space of $X.$ If $Y$ is another matrix whose column space is the same as that of $X,$ then $H'=H.$

$\endgroup$
3
  • $\begingroup$ I do know that $Hu$ is the orthogonal projection of $u$ onto $col(X)$, but I think I'm confused how you proved that here. I understand that if $u$ is in the column space of $X$, then $u$ can be written as a linear combination of the columns of $X$ and that $Hu = u$. And if $u$ is is orthogonal to the column space of $X$, you get $Hu = 0$. But this only seems to consider the cases where $u$ is (1) orthogonal to columns of $X$ (2) lies in the column space of $X$. But what about the case where $u$ is neither orthogonal nor does it lie entirely in the column space of $X$? $\endgroup$
    – 24n8
    May 5 at 13:49
  • $\begingroup$ I'm a bit rusty on this subject, but I think I may get it. Let's define $u, v$ to lie in the column space of $X$, to be orthogonal to the column space of $X$, respectively. And suppose $w = u + v$, which is the case I was wondering about in the previous comment. So we have $Hw= Hu + Hv = Hu = u$. So this shows that for some generic vector $w$, when the projection matrix operates on it, it creates $u$, which lies in the column space of $X$, and by definition of $w = u+v$ (two orthogonal components), we get that $Hw$ is the orthogonal projection of $w$ onto $X$. $\endgroup$
    – 24n8
    May 5 at 13:53
  • $\begingroup$ @anonuser01 : Every vector that is neither in the column space of $X$ nor orthogonal to the column space of $X$ is the sum of one vector that is in the column space of $X$ and another that is orthogonal to the column space of $X.$ And in only one way. $\qquad$ $\endgroup$ May 6 at 1:42
0
$\begingroup$

$\def\o{{\tt1}}$Let $\{e_k\}$ denote the standard vector basis, then the all-ones vector is $$\sum_{k=1}^ne_k=\o$$ The hat matrices for $X$ and $Y$ are $$H_x=X(X^TX)^{-1}X^T=XX^+\\H_y=Y(Y^TY)^{-1}Y^T=YY^+$$ Invoking the first Penrose condition of the pseudoinverse and the fact that $\o$ is the first column of $X$ yields these interesting results $$\eqalign{ H_x\o &= XX^+(\o) = XX^+(Xe_1) = Xe_1 = \o \\ Ye_1 &= (X-\o c^T)e_1 = \o-(c^Te_1)\o = \gamma\o \\ H_y\o &= (YY^+)(\gamma^{-1}Ye_1) = \gamma^{-1}Ye_1 = \o \\ }$$ Assume that $u_x$ lies in the column space of $X$, then for an arbitrary $b$ vector $$\eqalign{ u_x &= Xb \\ H_xu_x &= X(X^TX)^{-1}X^T(Xb) = Xb = u_x \\ }$$ Likewise, if $u_y$ lies in the column space of $Y$ then (utilizing the same $b$ vector) $$\eqalign{ u_y &= Yb \qquad\implies\quad H_yu_y = u_y \\ u_y &= \left(X-\o c^T\right)b \;=\; \big(u_x - \alpha\o\big) \\ }$$ Now compute the action of the hat matrix $H_x$ on $u_y$ $$\eqalign{ H_xu_y &= H_xu_x - \alpha H_x\o \\ &= u_x-\alpha\o \\ &= u_y \\ &= H_yu_y \\ }$$ So the action of $H_x$ on $u_y$ is identical to that of $H_y$

Similarly, the action of $H_y$ on $u_x$ is seen to be identical to that of $H_x$ $$\eqalign{ H_yu_x &= H_y(u_y+\alpha\o) \\ &= u_y + \alpha\o \\ &= u_x \\ &= H_xu_x \\ }$$ Therefore the matrices $H_x$ and $H_y$ are identical.


Note that the constraint $$c_1 = c^Te_1 = (1-\gamma) = 0 \quad\implies\quad \gamma=1$$ was not utilized in this analysis. In fact, the above analysis assumes that $$\gamma\ne 0 \quad\implies\quad c_1\ne 1$$

This is basically Michael Hardy's answer, but with more algebra.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.