Let $X$ be an $m \times n$ matrix. Define the orthogonal projection onto the column space of $X$ as


Also, define the linear span of a set of $m\times 1$ column vectors of $X$ as $\mbox{span}(X)$.

I'm wondering what the relation between $P(X)$ and vector space $\mbox{span}(X)$ is. Can you help me with an intuitive explanation? Intuitive and easy interpretation would be appreciated.

  • $\begingroup$ I am not following some things. $X$ gives us $n$ vectors of $\mathbb{R}^m$. We define the projection on $\mathbb{R}^m$ ? and we define the projection over the span of the columns of $X$ ? So I suppose $n\leq m$ ? Another thing, I don't catch what means $X'$. What do you refer as a set of $mx1$ vectors, which vectors? $\endgroup$ – HFKy Jul 20 '18 at 2:07
  • $\begingroup$ First, $X'$ is the transpose of $X$. It's assumed that $n \leq m$ holds. $m\times 1$ vectors refer to each $m\times 1$ columns of $X$. $\endgroup$ – user568810 Jul 20 '18 at 2:37
  • $\begingroup$ $m$x$1$ columns ? $X$ doesn't have $n$ column vectors, and $n \leq m ?$ $\endgroup$ – HFKy Jul 20 '18 at 2:39
  • $\begingroup$ Consider EACH column of $X$. $X$ has $n$ number of $m\times 1$ vectors. And yes, $n\leq m$. $\endgroup$ – user568810 Jul 20 '18 at 2:41
  • $\begingroup$ "mx1 vectors" ???. $X$ is mxn is impossible to get "mx1 vectors" whatever it means if $n \leq m$. Being $X$ mxn you have $n$ column vectors of $\mathbb{K}^m$ meaning that each of these vectors have $m$ components. $\endgroup$ – HFKy Jul 20 '18 at 2:45

You have used the "regression" tag, so I assume that the context in linear regression.

The columns of design matrix $X$ form a vector space if there is no intercept in the model $Y= X\beta$, in a case of an intercept this is an affine space. The intuitive relation is that the hat matrix $H = X(X'X)^{-1}X'$ projects the $n$ dimensional response vectors $y$ into the space that is spanned by your explanatory variables. Namely, $Hy=\hat{y}$ gives you the "closest" vector that can be uniquely represented by a linear combination of the columns of $X$ (explanatory variables).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.