There's a question in a textbook I've just looked at. Let $A$ be an $m\times n$ matrix. Prove that for all $\mathbf{b}\in \mathbb{R}^m$ there exists $\hat{\mathbf{x}}\in \mathbb{R}^n$ for which $A^T(A\hat{\mathbf{x}}-\mathbf{b}) = \mathbf{0}$.
Hint: $\text{col}(A)+\text{col}(A)^{\perp} = \mathbb{R}^m$.
Just wondering how to go about it. Thanks!

  • 3
    $\begingroup$ Fun trivia: This calculation lies at the core of least-squares linear regression. The $\hat{\mathbf x}$ you find here is the least-square-error "solution" to the potentially inconsistent equation $A\mathbf x=\mathbf b$. $\endgroup$ – Arthur Aug 14 at 7:41
  • $\begingroup$ Choose $\hat x$ so that $A \hat x$ is as close as possible to $b$. Visually, the residual $r = b - A \hat x$ most be orthogonal to the column space of $A$. In particular, $r$ is orthogonal to each column of $A$. (This is the key intuition behind the normal equations for least squares problems.) $\endgroup$ – littleO Aug 14 at 7:44
  • $\begingroup$ So we are just deriving $A^TA\hat{\mathbf{x}} = A^T\mathbf{b}$ thanks. Just wasn’t sure how to use the hint or even if we have to! $\endgroup$ – squenshl Aug 14 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.