Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Given a matrix $A \in R^{m \times n}$ and whose rank is $n$. I need to show $\| A(A^TA)^{-1}A^T\|_2 = 1$. Can any hint me the direction in which I should solve this problem. Should I use any decomposition of matrix $A$ to show the result?

share|improve this question
    
You can use either QR or SVD for decomposing your matrix. –  J. M. Sep 6 '11 at 5:37
    
I suppose I should have asked: are you familiar with either of those decompositions? –  J. M. Sep 6 '11 at 5:46
    
@J.M I learnt about eigen value decomposition but neither SVD nor QR decomposition –  Learner Sep 6 '11 at 5:55
    
How about QR?$\;$ –  J. M. Sep 6 '11 at 5:56
    
@j.M. Jus edited my previous comment –  Learner Sep 6 '11 at 5:58
show 3 more comments

4 Answers 4

up vote 6 down vote accepted

I will explain it from a geometry perspective.

Let $P=A(A^TA)^{-1}A^T$. $P$ actually is an orthogonal projection matrix satisfying $P^2=P$ and $P^T=p$. Let $x\in\mathbb{R}^m$. Then $Px$ is the orthogonal projection of $x$ in the column space of $A$.

Consider an orthogonal basis $\{x_i\}_{i=1}^n$ of $\mathrm{Range}(A)$. Then $Px_i=x_i$. Consider an orthogonal basis $\{y_i\}_{i=1}^{m-n}$ of the orthogonal complement of $\mathrm{Range}(A)$, then $Py_i=0$.

So the matrix $P$ has $n$ eigenvalues as 1 and $m-n$ eigenvalues as 0. If $\|*\|_2=\sigma_{\max}(*)$, of course $\|P\|_2=1$.

share|improve this answer
add comment

An argument without geometry goes like this:

As Shiyu said $P^2=P$ and hence, $\|P\| = \|P^2\|\leq \|P\|^2$ and therefor $1\leq \|P\|$. Moreover, $P^T=P$ and hence, $\|Px\|^2 = \langle Px,Px\rangle = \langle Px, x\rangle \leq \|Px\|\|x\|$ which gives $\|P\| \leq 1$.

share|improve this answer
add comment

Let $H = A(A^TA)^{-1}A^T$. To see that $x\mapsto Hx$ (for $x\in\mathbb{R}^m$) is the orthogonal projection onto the column space of $A$, it suffices to prove two things:

  • If $x$ is in the column space of $A$, then $Hx=x$.
  • If $x$ is orthogonal to all columns of $A$, then $Hx=0$.

To prove the second statement, notice that if $x$ is orthogonal to all columns of $A$, then $A^T x = 0$. Therefore $A(A^TA)^{-1}A^Tx = 0$.

To prove the first statement, notice that $x$ is in the column space of $A$ iff $x = Aw$, for some $w$. Therefore $$ Hx = HAw = \Big(A(A^TA)^{-1}A^T\Big) Aw = A(A^TA)^{-1}\Big(A^T A\Big)w = Aw = x. $$

Now let $x$ be any vector in $\mathbb{R}^m$. Decompose $x$ into a component in the column space of $A$ and a component orthogonal to the column space of $A$. The component in the column space of $A$ is $u=Hx$. The component orthogonal to the column space of $A$ is $v=(I-H)x$. What then is $\|Hx\|_2$? It is $\|u\|_2 \le \|u+v\|_2 = \|x\|_2$. Since $\|Hx\|_2 \le \|x\|_2$, we have $\|H\|_2 \le 1$. But since $\|Hu\|_2= \|u\|_2$, we have $\|H\|_2\ge1$.

share|improve this answer
    
BTW, it is conventional in statistics to use the letter $H$ for this matrix and to call it the "hat matrix" (hence the "$H$") because of the meaning it has in statistics. –  Michael Hardy Sep 7 '11 at 14:07
add comment

I'm probably missing something, but since $(AB)^{-1}=B^{-1}A^{-1}$, why not just

$$A(A^TA)^{-1}A^T = A(A^{-1}(A^T)^{-1})A^T = (AA^{-1})((A^T)^{-1}A^T) = II = I$$ ?

share|improve this answer
3  
$A$ is usually not square, so... –  J. M. Sep 6 '11 at 13:39
    
Oh, of course... –  Henning Makholm Sep 6 '11 at 13:41
1  
"Pure" mathematicians often tacitly assume matrices are square, it seems. –  Michael Hardy Sep 6 '11 at 13:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.