I was looking at the derivation of the normal equation from here.

Now, the author has used the fact that $A^{\mathrm T} B = B^{\mathrm T} A$ to reach the step shown in the below image. Can anyone provide some information like, when is it true, or how we can prove it?

$$J(\theta) = ((X\theta)^{\mathrm T} -y^{\mathrm T})(X\theta -y)$$ $$J(\theta) = (X\theta)^{\mathrm T} X\theta -\color{blue}{(X\theta)^{\mathrm T} y \color{black}{-} y^{\mathrm T} (X\theta)} +y^{\mathrm T} y$$ Note that $X\theta$ is a vector, and so is $y$. So when we multiply one by another, it doesn't matter what the order is (as long as the dimensions work out). So we can further simplify: $$J(\theta) = \theta^{\mathrm T} X^{\mathrm T} X \theta -\color{blue}{2(X\theta)^{\mathrm T} y} +y^{\mathrm T} y$$

  • 1
    $\begingroup$ $(AB)^T=B^TA^T$ $\endgroup$
    – Botond
    Apr 25, 2018 at 13:51

2 Answers 2


Note that $(A^{\rm T} B)^{\rm T} = B^{\rm T} (A^{\rm T})^{\rm T} = B^{\rm T} A$, so if you are constraining $A^{\rm T} B = B^{\rm T} A$, that implies $(A^{\rm T} B)^{\rm T} = A^{\rm T} B$, meanining $A^{\rm T} B$ is symmetric.

  • $\begingroup$ Got the symmetric part...So does it mean that $(X\theta)^Ty$ will always be a symmetric matrix?...how to prove that w.r.t normal equation derivation? $\endgroup$
    – Sourajit
    Apr 25, 2018 at 14:30
  • 2
    $\begingroup$ Is $X$ a matrix and both $\theta$ and $y$ are column vectors? If so, then $(X\theta)^{\rm T} y$ will always be symmetric because $X\theta$ is a vector and $(X\theta)^{\rm T} y$ is a scalar (a $1\times 1$ matrix), and scalars are always symmetric. $\endgroup$
    – Rócherz
    Apr 25, 2018 at 14:35

Let's call $$C=AB$$ Then we have that $$C_{ij}=A_{ik}B_{kj}$$ We know that $$(M^T)_{ab}=M_{ba}$$ So we have that: $$(C^T)_{ij}=C_{ji}$$ $$(C^T)_{ij}=A_{jk}B_{ki}$$ $$(C^T)_{ij}=B_{ki}A_{jk}$$ $$(C^T)_{ij}=(B^T)_{ik}(A^T)_{kj}$$ And finally: $$C^T=B^TA^T$$ $$(AB)^T=B^TA^T$$


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .