I have something like this $$X^T A^T A X - X^T A^T B - B^T A X$$ or $$X^T A^T A X -2 X^T A^T B$$ since X and B are really vectors ($X^TA^TB=B^TAX$).

I’d like to find the X that minimizes the expression. To do that I want to have something like this $$[X-(A^TA)^{-1}A^TB]^T(A^TA)[X-(A^TA)^{-1}A^TB]-B^TA(A^TA)^{-1}A^TB$$ so that the X that minimizes the expression is $(A^TA)^{-1}A^TB$.

My question is: how does all this relate to the technique of completing the squares? How can I show the squares?

To further explain my question, it’s easy to say, if you have $a^2 + 2ab$ just add and subtract $b^2$. But how can I abstract that to matrices?

PS: I assume $A^TA$ is non singular, positive definite.

  • $\begingroup$ (i) What is "C"? Is the C in line 5 supposed to be B? (ii) Since matrix multiplication is not commutative, $(A+B)^2 = A^2 + AB + BA + B^2$. If you have $A^2+2AB$, simply adding and subtracting $B^2$ will not give you $(A+B)^2-B^2$. $\endgroup$ Jun 12, 2011 at 4:33
  • $\begingroup$ As written, the first equation does not make sense. The first term is a scalar, but the other terms are vectors, presuming that $A$ and $B$ are matrices. Does the missing $C$ fix that? In truth, none of equations make sense for the same reason. $\endgroup$
    – rcollyer
    Jun 12, 2011 at 4:39
  • $\begingroup$ Sorry I’ll fix the question. $\endgroup$
    – gurghet
    Jun 12, 2011 at 4:42
  • $\begingroup$ In my question there is not a problem to be solved, more like a solution to be explained. Going to the first equation to the second equation (the long one) causes me much trouble. $\endgroup$
    – gurghet
    Jun 12, 2011 at 5:07

1 Answer 1


Write $$X^T A^T A X - X^T A^T B - B^T A X = (A X - B)^T (A X - B) - B^T B.$$

So you want to minimize $(A X - B)^T (A X - B)$.

Note that $A (A^T A)^{-1} A^T$ is the orthogonal projection on ${\rm Ran}(A)$. Thus $B = A u + v$ where $u = (A^T A)^{-1} A^T B$, and $v \perp A x$ for all $x$.

Now $A X - B = A(X - u) - v$, so $(A X - B)^T (A X - B) \ge v^T v$. Thus the minimum value of $(A X - B)^T (A X - B)$ is $v^T v$, attained for $X = u$.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .