In many books and articles I sometimes see similar matrix constructs, multiplication of the matrix transpose (or inverse), some other matrix, and then the first matrix itself, like this: $M^T AM$ or $M^{-1} A M$.

For square matrices $M$ I've found an example of definition of matrix congruence, that is, existence of certain matrices $A,B,P$ related so that $P^T A P = B$ names $A$ and $B$ congruent. However, this only applies to square $A,B$ and invertible $P$. Also I'm not aware of practical usage of this concept.

Again, for square $A, B, P$ there exists a concept of matrix similarity, that is, relation of $B=P^{-1}AP$ names $A$ and $B$ similar. I'm more familiar with this concept since it's frequently used in computer graphics when $B$ acts as "base" model-view matrix, and series of $P_i$ act as a set of modelling matrices for series of objects in a scene.

However, these "tricks" for non-square matrices are also frequently used in literature, for example, in normal equations for non-linear least squares methods and algorithms.

The question is: what are the informal explanations of $M^T AM$ or $M^{-1} A M$ matrix "tricks" and how they are used in applications?

  • $\begingroup$ if you know a pinch of group theory, and e.g. A is invertible, then $M^{-1} A M$ is conjugating $A$. The congruence transform $M^T A M$ makes perfect sense in context of bilinear forms. Sylvester's Law of Interita and e.g. $LDL^T$ or even Cholesky factorization can be interpretted as applications of Sylvester's Law. $\endgroup$ May 31 '20 at 3:27

There’s no “trick” here. These all represent a change of basis (coordinate system) for various objects that the matrix $A$ might represent. The matrix $M$ converts coordinates of a vector from one basis to another.

When $A$ represents a linear transformation, the appropriate formula is $M^{-1}AM$. The matrix $M$ converts from some coordinate system into the one that $A$ is expecting, and $M^{-1}$ converts its output back into the original coordinate system. Algebraically, if we have $x=Mx'$ and $y=My'$, then from $y=Ax$ we have $My'=AMx'$, and multiplying both sides by $M^{-1}$ produces $y'=(M^{-1}AM)x'$.

Similarly, $M^TAM$ is a change of basis when $A$ represents a bilinear or quadratic form. Here, we’re working with an expression of the form $x^TAy$, from which $x^TAy = (Mx')^TA(My) = x'^T(M^TAM)y$. You might also see $M^TAM$ when $A$ represents a linear transformation, but that’s a special case of $M^{-1}AM$ when $M^{-1}=M^T$, which occurs when the change of basis is achieved via an orthogonal transformation.

  • $\begingroup$ But what about the cases where $M^T A M$ is used with non-square $M$? In that case $A$ has to be square anyways to satisfy the multiplication rules, but $M$ doesn't seem to represent any meaningful change of basis. I've seen this in LSM method for over-determined set of points. $\endgroup$
    – mbaitoff
    May 31 '20 at 11:11
  • $\begingroup$ @mbaitoff It all really depends on context—you have to know what the matrices represent. It could still be a change of basis, but the basis is for a subspace of the ambient space. Or, $A$ might be the Gram matrix of an inner product, in which case $M^TAM$ computes all of the pairwise inner products of the columns of $M$ “in bulk.” $\endgroup$
    – amd
    May 31 '20 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.