0
$\begingroup$

Solving the problem: $A\vec{x}=\vec{b}$. I'm considering $\mathbb{R}^3$ and the span of $A$ being two dimensional, and $\vec{b}$ not being in the span of $A$, but I think my intuition holds for any vector space.

Minimising the length of the error vector is a big name for finding the closest image of $A$ from $\vec{b}$, which looks geometrically like projecting $\vec{b}$ and the plane of the span of $A$, and then solving $A\vec{x}=\vec{b}_{proj}$.

What I am looking for is a proof that the formula $(A^T A)^{-1} A^T\vec{b}$ works, which would highlight those geometrical intuitions. What I was hoping for was to find one part on the right of the solution representing the the projection, and the other right part representing the "inverse" of $A$, but I failed to find any link.

$\endgroup$
2
  • $\begingroup$ are you familiar with the pseudoinverse? $\endgroup$ – tch Oct 25 '19 at 3:38
  • $\begingroup$ If $x$ is chosen so that $Ax$ is as close as possible to $b$, then visually the residual $r = b - Ax$ is orthogonal to the column space of $A$. In particular, $r$ is orthogonal to each column of $A$. In other words, $A^T r = A^T(b - Ax)=0$. This is the visual meaning of the normal equations. $\endgroup$ – littleO Oct 25 '19 at 8:50
1
$\begingroup$

I assume that $A$ has full rank and more rows than columns. Otherwise the notation $(A^T A)^{-1}$ would not make sense. In the OP's example, this means $A\in\mathbb{R}^{3\times 2}$ and $\vec{x}\in\mathbb{R}^2$.

$A\vec{x}=\vec{b}_{\mathrm{proj}}$ is another way of saying that the line through $\vec{b}$ and $A\vec{x}$ is perpendicular to the span of $A$. Therefore, $$ \langle \vec{v} , A\vec{x}-\vec{b} \rangle = 0 \;\;\forall\;\vec{v}\in\mathrm{span}(A) $$ or, even simpler, $$ A^T (A\vec{x} - \vec{b}) = 0 $$ which immediately leads us to $$ \vec{x} = (A^T A)^{-1}A^T\vec{b} $$

$\endgroup$
1
  • $\begingroup$ Oh, of course! How could I not have thought about it? Thank you. $\endgroup$ – Jonas Daverio Oct 25 '19 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.