Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have something like this $$X^T A^T A X - X^T A^T B - B^T A X$$ or $$X^T A^T A X -2 X^T A^T B$$ since X and B are really vectors ($X^TA^TB=B^TAX$).

I’d like to find the X that minimizes the expression. To do that I want to have something like this $$[X-(A^TA)^{-1}A^TB]^T(A^TA)[X-(A^TA)^{-1}A^TB]-B^TA(A^TA)^{-1}A^TB$$ so that the X that minimizes the expression is $(A^TA)^{-1}A^TB$.

My question is: how does all this relate to the technique of completing the squares? How can I show the squares?

To further explain my question, it’s easy to say, if you have $a^2 + 2ab$ just add and subtract $b^2$. But how can I abstract that to matrices?

PS: I assume $A^TA$ is non singular, positive definite.

share|cite|improve this question
(i) What is "C"? Is the C in line 5 supposed to be B? (ii) Since matrix multiplication is not commutative, $(A+B)^2 = A^2 + AB + BA + B^2$. If you have $A^2+2AB$, simply adding and subtracting $B^2$ will not give you $(A+B)^2-B^2$. – Arturo Magidin Jun 12 '11 at 4:33
As written, the first equation does not make sense. The first term is a scalar, but the other terms are vectors, presuming that $A$ and $B$ are matrices. Does the missing $C$ fix that? In truth, none of equations make sense for the same reason. – rcollyer Jun 12 '11 at 4:39
Sorry I’ll fix the question. – gurghet Jun 12 '11 at 4:42
In my question there is not a problem to be solved, more like a solution to be explained. Going to the first equation to the second equation (the long one) causes me much trouble. – gurghet Jun 12 '11 at 5:07
up vote 4 down vote accepted

Write $$X^T A^T A X - X^T A^T B - B^T A X = (A X - B)^T (A X - B) - B^T B.$$

So you want to minimize $(A X - B)^T (A X - B)$.

Note that $A (A^T A)^{-1} A^T$ is the orthogonal projection on ${\rm Ran}(A)$. Thus $B = A u + v$ where $u = (A^T A)^{-1} A^T B$, and $v \perp A x$ for all $x$.

Now $A X - B = A(X - u) - v$, so $(A X - B)^T (A X - B) \ge v^T v$. Thus the minimum value of $(A X - B)^T (A X - B)$ is $v^T v$, attained for $X = u$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.