Given three matrices $A$, $B$ and $C$ such that $ABA^T C$ is a square matrix, the derivative of the trace with respect to $A$ is:

$$ \nabla_A \operatorname{trace}( ABA^{T}C ) = CAB + C^T AB^T $$

There is a proof here, page 4 (PDF file). However, I was not able to understand the second line of the proof. it says something like this:

$$ \nabla_A \operatorname{trace}( ABA^T C )= \nabla_\bullet \operatorname{trace}(f(\bullet)A^{T}C) + \nabla_\bullet \operatorname{trace}(f(A) \bullet^T C) $$

Where $f(A) = AB$

I tried searching the proof or any hint using google, but I couldn't fine anything. Could anyone please help me with this second line. I cannot sleep. I am obsessed with this proof.

  • $\begingroup$ I am always confused by these matrix derivatives so I have found that the most solid way, for my own mental health, is working by expanding all components, as here: math.stackexchange.com/a/2302685/259671 $\endgroup$ – Miguel Sep 27 '17 at 18:38
  • $\begingroup$ Unfortunately, the link's down. $\endgroup$ – Viktor Glombik Mar 16 '19 at 14:22

Since those "dots" appear nowhere else in the document, and the calculation is labeled "funky," I wonder if the author himself even understands what is going on there. (If I came across this while grading, I'd assume the writer had asked a friend for an explanation and copied down only half of what they said. It's a level of detail associated more with personal notes than communicating with others.)

I don't mean to criticize: the spirit of that document is to collect a number of matrix calculations and that is fine. But when the author uses the word "prove" (in "we prove that...") he is using it in a very loose sense. Enough detail is given in the calculation so that if you understand it, you could go and construct a proof. But a chain of unjustified, unexplained equations is not a "proof" in the strict sense. (The appearance of "$f'$" in that context cannot actually be explained in a way consistent with his definitions of $f$ and $'$, and so maybe counts as a mistake--- although it is immediately replaced with the right thing.)

Anyway, it is implicit use of the chain rule. ("Implicit" in the English language sense of "there but not mentioned", not in any sense like implicit differentiation.) Here is a scalar-calculus analogy. One way to compute the derivative of $$ g(a) = a^a $$ is to regard $g(a)$ as the result of plugging the pair of functions $x = a$ and $y = a$ into the multivariable function $H(x,y) = x^y$. Whenever $H(x,y)$ is a function of two variables $x$ and $y$, each of which is a function of $a$, the multivariable chain rule tells you that $$ \frac{d}{da} H(x,y) = \frac{\partial H(x, y)}{\partial x} \cdot \frac{dx}{da} + \frac{\partial H(x,y)}{\partial y} \cdot \frac{dy}{da} $$ and the the partials of $H(x,y) = x^y$ are easy to calculate since in computing the $x$-partial derivative you hold $y$ fixed, and vice versa: $\frac{\partial H}{\partial x} = y x^{y-1}$ (from the power rule) and $\frac{\partial H}{\partial y} = x^y \ln(x)$ (from the rule for differentiating exponential functions). So with $H(x,y) = x^y$ and $x=a$ and $y=a$ in the above, we see \begin{align*} \frac{d}{da} (a^a) & = \frac{\partial H(a, a)}{\partial x} \cdot 1 + \frac{\partial H(a,a)}{\partial y} \cdot 1 \\ & = a a^{a - 1} + a^a \ln(a) = a^a + a^a \ln(a). \end{align*} It is not too much trouble to follow this calculation because I introduced new variables $x$ and $y$ and explicitly defined the function $H$ I was using. If instead of writing the new variables I wrote dots, and instead of defining my new function $H$ I just wrote out formulas with $a$s and dots, it might get confusing.

That's basically what's going on in that document. Fix matrices $B$ and $C$. The function $$ g(A) = \operatorname{trace} (ABA^TC) $$ of the matrix $A$ has the form $$ g(A) = H(f(A),A) $$ for the scalar valued function $H(X,Y) = \operatorname{trace} X Y^T C$ of two matrix variables and the matrix valued function $f(A) = AB$ of one variable. The chain rule implies that in this case we have $$ \nabla_A H(f(A),A) = (\nabla_X H(X,Y)) B^T + \nabla_Y H(X,Y) $$ where you evaluate the right hand side at $X = f(A)$ and $Y = A$ after computing the matrix derivatives. Elsewhere in that document he sketches how to compute $\nabla_X H(X,Y) = (Y^T C)^T$ and $\nabla_Y H(X,Y) = CX$. So that is how the result $(A^T C)^T B^T + CAB$ comes out. Just like the scalar case, introducing new variables so that you can treat some as constant when differentiating with respect to others, simplifies the calculation.

To return to my initial remark about $f'$: many of that document's formal particulars are sketchy. For example, the nonrigorous treatment of the Jacobian in section 4 is just to motivate calculation, and the appearance of $f'$ in the calculation you ask about is not consistent with this use. My guess is that the author wanted to remind himself that something coming from the chain rule and related to $f$ went there, and so he wrote $f'$ as a mental reminder. Since he replaced it with the right thing, $B^T$, no harm was done.

... And one is right to be cautious about what exactly goes there, since for arbitrary matrix valued functions $X = f(A)$ and $Y = k(A)$, the expression for $\nabla_A H(f(A),k(A))$ is a complicated function of the entries of $\nabla_X H(X,Y)$ and $\nabla_Y H(X,Y)$ and the partial derivatives of the entries of $f(A)$ and $g(A)$. It is not easily written as a matrix product of $\nabla_X H(X,Y)$ with a matrix depending only on $f$, plus a matrix product of $\nabla_Y H(X,Y)$ with a matrix depending only on $g$. We got a simple formula above because our $f(A) = AB$ and $k(A) = A$ were simple functions of $A$ (e.g. any one entry of $f(A)$ depends only on one row of entries of $A$, and not all of the entries of $A$, as might happen in general). If you were wondering why the author did not include a general matrix derivative chain rule, this is why. It doesn't look very nice in general; it only looks nice in various special cases.

  • $\begingroup$ Hi leslie. Now, I understand that proof. Your explanations was very clear. I didn't see matrix calculus and I think the matrix notation was confusing me. Thank you so much. $\endgroup$ – Harold Apr 29 '12 at 23:58
  • 2
    $\begingroup$ In the following line I am assuming that $B^T$ is the derivative of $f(A)$ w.r.t $A$. $$ \nabla_A H(f(A),A) = (\nabla_X H(X,Y)) B^T + \nabla_Y H(X,Y) $$But $f(A)$ is a matrix valued function. To my understanding the derivative of $tr(f(A))$ would have been $B^T$. I don't know what should be the case here..... $\endgroup$ – Shirin Jul 1 '13 at 9:54
  • $\begingroup$ @Shirin Yeah, I have the same question here. To my understanding, $\nabla$ operator only takes in a function that maps a matrix to a real number. It is not allowed to take in a matrix directly. $\endgroup$ – SundayCat Jul 31 '17 at 21:09
  • $\begingroup$ I have the same question as @Shirin and SumdayCat. Expanding the derivation a bit, I don't think the appearance of $B^T$ is explained. It is $\nabla_A tr(X) = B^T$, NOT $\nabla_A X$. No idea what $\nabla_A Y$ should be either. I wonder if the multivariate derivative is correctly applied to the matrix case here. \begin{align*} \nabla_A H(f(A),A) &= \nabla_A H(X, Y) \\ &= (\nabla_X H(X,Y)) \nabla_A X + (\nabla_Y H(X,Y)) \nabla_A Y \\ ?&= (\nabla_X H(X,Y)) B^T + \nabla_Y H(X,Y) \\ \end{align*} $\endgroup$ – zyxue Sep 27 '17 at 17:04

$\nabla$ is not the derivative but the gradient of a function $f$, that is $(\nabla f)(x) \cdot v=Df_x(v)$ where $\cdot$ is a dot product and $Df_x(v)$ is the derivative in $x$ in the direction of the vector $v$. For the real matrices, we consider the following scalar product:

$$(U,V)\rightarrow U \cdot V=\text{Tr}(U^TV).$$

Here the derivative is \begin{align*} Df_A:H &\rightarrow \text{Tr}(HBA^TC)+\text{Tr}(ABH^TC) \\ &= \text{Tr}(BA^TCH)+Trace(B^TA^TC^TH) \\ &= \text{Tr}((C^TAB^T+CAB)^TH) \\ &= (C^TAB^T+CAB) \cdot H. \end{align*}

Finally $(\nabla f)(A)=C^TAB^T+CAB$ as required.

  • $\begingroup$ Mallesh, that you write below is a non-sense. The derivative in $A$ of $f:A\rightarrow AB$ is $Df_A:H\rightarrow HB$ and $Df_A=I\otimes B$, a Kronecker product. $\endgroup$ – loup blanc Aug 30 '13 at 16:49
  • $\begingroup$ Certainly, you'd want to mention the function $g:A\rightarrow Tr(AB)$. Then $Dg_A:H\rightarrow Tr(HB)=B^T.H$ and $(\nabla g)(A)=B^T$. $\endgroup$ – loup blanc Aug 31 '13 at 7:48
  • $\begingroup$ Just added some explanation to the equations: 1) multivariate derivative, in general: $\partial XY = (\partial X) Y + X (\partial Y)$ 2) left: because $tr(XY) = tr(YX)$; right because $tr(X) = tr(X^T)$ 3) because $tr(XZ) + tr(YZ) = tr((X + Y)Z)$ 4) Besides author's justification with dot product. $\endgroup$ – zyxue Sep 27 '17 at 18:02

I like your presentation and explanation very much. I am finding it difficult to follow how derivative of $AB$ with respect to $A$ turned out to be $B^T$. I am under the assumption that $f(A)$ is a function $f:R^{m X n} \rightarrow R$. Then $\nabla_A f(A)$ is defined and is an $m X n$ matrix with $(i,j)th$ entry being equal to $\frac{\partial }{\partial a_{i,j}}f(A) $.

$trace(ABA^TC)$ is a real valued function where as $AB$ is a matrix.


I recently came across the same problem from CS229 lecture notes on page 9. After learning from the other answers, I formulated my own derivation, which I think is straightforward.

I assume you've already proved the following, which are in the lecture notes, too.

\begin{align*} \mathrm{tr}(AB) &= \mathrm{tr}(BA) \\ \mathrm{tr}(A) &= \mathrm{tr}(A^T) \\ \nabla_A \mathrm{tr}(AB) &= B^T \\ \end{align*}

Then, using the multivariate derivative rule, I label the variate being looked at with $X$,

\begin{align*} \nabla_A \mathrm{tr}( ABA^T C ) &= \nabla_X \mathrm{tr}( XBA^T C ) + \nabla_X \mathrm{tr}( ABX^T C) \\ &= (BA^TC)^T + \nabla_X \mathrm{tr}(CABX^T) \\ &= C^TAB^T + \nabla_X \mathrm{tr}(X(CAB)^T) \\ &= C^TAB^T + ((CAB)^T)^T \\ &= C^TAB^T + CAB \\ \end{align*}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.