Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

In my work, I have repeatedly stumbled across the matrix (with a generic matrix $X$ of dimensions $m\times n$ with $m>n$ given) $\Lambda=X(X^tX)^{-1}X^{t}$. It can be characterized by the following:

(1) If $v$ is in the span of the column vectors of $X$, then $\Lambda v=v$.

(2) If $v$ is orthogonal to the span of the column vectors of $X$, then $\Lambda v = 0$.

(we assume that $X$ has full rank).

I find this matrix neat, but for my work (in statistics) I need more intuition behind it. What does it mean in a probability context? We are deriving properties of linear regressions, where each row in $X$ is an observation.

Is this matrix known, and if so in what context (statistics would be optimal but if it is a celebrated operation in differential geometry, I'd be curious to hear as well)?

share|improve this question
1  
Perhaps you mean $X(X^TX)^{-1}X^T$? Otherwise the sizes don't match. –  Rahul Jun 23 '11 at 7:58
1  
If so, I think the properties you have listed are a perfectly good interpretation: the matrix $\Lambda$ projects any vector $v$ to the column space of $X$. –  Rahul Jun 23 '11 at 8:00
    
Thanks. Typo fixed. –  Har Jun 23 '11 at 8:04
1  
Oh, I actually figured it out now. $(I-\Lambda)$ takes outcomes $Y$, does a regression and spits out the estimated residuals. If anyone else is interested, I'll elaborate more. –  Har Jun 23 '11 at 8:15
add comment

2 Answers

up vote 11 down vote accepted

It is also called hat matrix. The idea is that this matrix "gives the hat": transforms the dependent variable to its prediction in linear regression.

The linear regression model is the following:

$$y=X\beta+\varepsilon.$$

The least squares estimate of the $\beta$ is defined as

$$\hat\beta=(X^TX)^{-1}X^Ty.$$

The prediction of the model is then:

$$\hat{y}=X\hat\beta=X(X^TX)^{-1}X^Ty$$

So we get that matrix $X(X^TX)^{-1}X^T$ transforms $y$ to $\hat{y}$, hence the hat matrix.

share|improve this answer
    
Beautiful. This was exactly what I was looking for. I just (well, 35 minutes ago) managed to deduce the relevant properties but knowing that it is called the hat matrix might be very useful! Thanks! –  Har Jun 23 '11 at 8:52
add comment

This should be a comment, but I can't leave comments yet. As pointed out by Rahul Narain, this is the orthogonal projection onto the column space of $X$

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.