# How to prove that least squares projection matrix is invariant to constant shifts in $X$?

Suppose you have a matrix $$X \in \mathbb{R}^{n \times p}$$ and the projection matrix in least squares is defined as:

$$H = X(X^TX)^{-1}X^T$$

where note that the first column of $$X$$ are ones.

Now suppose that you define $$Y = X - \boldsymbol{1}_{n\times 1} C^T$$ where $$C \in \mathbb{R}^{p \times 1}$$ and $$C_1 = 0$$. So $$Y$$ is defined as $$X$$ shifted by some constant amount, $$C_i$$ subtracted from the $$i$$-th column of $$X.$$

How do we show that

$$H' = Y(Y^TY)^{-1} Y^T = H$$

Here is my attempt (i'm going to drop the bold on the column vector of 1s): $$H' = Y(Y^TY)^{-1} Y^T \\ = (X - 1C^T)((X - 1C^T)^T(X - 1C^T))^{-1}(X - 1C^T)^T \\ = (X - 1C^T)((X^T - C1^T)(X - 1C^T))^{-1}(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)^{-1}(X^T - C1^T) \\$$

I stopped here because I know this is going to get messy and will probably involve block matrices, and I think there should be an easier solution than what I'm undertaking?

I believe there's an argument that can be made about how $$H$$ is spanned by the same basis before and after shifting since $$X$$ is defined to have a column of ones, so any column of $$Y$$ can be written as the corresponding column of $$X$$ and the first column of $$X$$.

• This will come down to the fact that the column spaces of $H$ and $H'$ are identical. May 5 at 3:43

The matrix $$X$$ in this scenario is typically tall and skinny, having many more rows than columns. The matrix $$X^TX$$ is invertible precisely if the columns of $$X$$ are linearly independent.

If $$u$$ is in the column space of $$X$$ then $$u=Xb$$ for some column vector $$b$$ with only as many entries as $$X$$ has columns, and then a simple algebraic computation shows that $$Hu=u.$$

But if $$u$$ is orthogonal to the column space of $$X$$ then $$X^Tu= 0$$ and so $$Hu=0.$$

Thus for any column vector $$u$$ as "tall" as $$X,$$ the vector $$Hu$$ is the orthogonal projection of $$u$$ onto the column space of $$X.$$

Therefore, the matrix $$H$$ depends on $$X$$ only through the column space of $$X.$$ If $$Y$$ is another matrix whose column space is the same as that of $$X,$$ then $$H'=H.$$

• I do know that $Hu$ is the orthogonal projection of $u$ onto $col(X)$, but I think I'm confused how you proved that here. I understand that if $u$ is in the column space of $X$, then $u$ can be written as a linear combination of the columns of $X$ and that $Hu = u$. And if $u$ is is orthogonal to the column space of $X$, you get $Hu = 0$. But this only seems to consider the cases where $u$ is (1) orthogonal to columns of $X$ (2) lies in the column space of $X$. But what about the case where $u$ is neither orthogonal nor does it lie entirely in the column space of $X$?
– 24n8
May 5 at 13:49
• I'm a bit rusty on this subject, but I think I may get it. Let's define $u, v$ to lie in the column space of $X$, to be orthogonal to the column space of $X$, respectively. And suppose $w = u + v$, which is the case I was wondering about in the previous comment. So we have $Hw= Hu + Hv = Hu = u$. So this shows that for some generic vector $w$, when the projection matrix operates on it, it creates $u$, which lies in the column space of $X$, and by definition of $w = u+v$ (two orthogonal components), we get that $Hw$ is the orthogonal projection of $w$ onto $X$.
– 24n8
May 5 at 13:53
• @anonuser01 : Every vector that is neither in the column space of $X$ nor orthogonal to the column space of $X$ is the sum of one vector that is in the column space of $X$ and another that is orthogonal to the column space of $X.$ And in only one way. $\qquad$ May 6 at 1:42

$$\def\o{{\tt1}}$$Let $$\{e_k\}$$ denote the standard vector basis, then the all-ones vector is $$\sum_{k=1}^ne_k=\o$$ The hat matrices for $$X$$ and $$Y$$ are $$H_x=X(X^TX)^{-1}X^T=XX^+\\H_y=Y(Y^TY)^{-1}Y^T=YY^+$$ Invoking the first Penrose condition of the pseudoinverse and the fact that $$\o$$ is the first column of $$X$$ yields these interesting results \eqalign{ H_x\o &= XX^+(\o) = XX^+(Xe_1) = Xe_1 = \o \\ Ye_1 &= (X-\o c^T)e_1 = \o-(c^Te_1)\o = \gamma\o \\ H_y\o &= (YY^+)(\gamma^{-1}Ye_1) = \gamma^{-1}Ye_1 = \o \\ } Assume that $$u_x$$ lies in the column space of $$X$$, then for an arbitrary $$b$$ vector \eqalign{ u_x &= Xb \\ H_xu_x &= X(X^TX)^{-1}X^T(Xb) = Xb = u_x \\ } Likewise, if $$u_y$$ lies in the column space of $$Y$$ then (utilizing the same $$b$$ vector) \eqalign{ u_y &= Yb \qquad\implies\quad H_yu_y = u_y \\ u_y &= \left(X-\o c^T\right)b \;=\; \big(u_x - \alpha\o\big) \\ } Now compute the action of the hat matrix $$H_x$$ on $$u_y$$ \eqalign{ H_xu_y &= H_xu_x - \alpha H_x\o \\ &= u_x-\alpha\o \\ &= u_y \\ &= H_yu_y \\ } So the action of $$H_x$$ on $$u_y$$ is identical to that of $$H_y$$

Similarly, the action of $$H_y$$ on $$u_x$$ is seen to be identical to that of $$H_x$$ \eqalign{ H_yu_x &= H_y(u_y+\alpha\o) \\ &= u_y + \alpha\o \\ &= u_x \\ &= H_xu_x \\ } Therefore the matrices $$H_x$$ and $$H_y$$ are identical.

Note that the constraint $$c_1 = c^Te_1 = (1-\gamma) = 0 \quad\implies\quad \gamma=1$$ was not utilized in this analysis. In fact, the above analysis assumes that $$\gamma\ne 0 \quad\implies\quad c_1\ne 1$$

This is basically Michael Hardy's answer, but with more algebra.