I am studying the matrix calculus page written in wikipedia and I have a question. In the table ' $\text{Identities: scalar-by-matrix}\frac{\partial y}{\partial \mathbf{X}}$ ' , it is shown :

(1) $\frac{\partial \operatorname{tr}(\mathbf{AX})}{\partial \mathbf{X}} = \frac{\partial \operatorname{tr}(\mathbf{XA})}{\partial \mathbf{X}} =\mathbf{A}^\top$ This can be easily proved by showing: $\frac{\partial}{\partial \mathbf{X_{ij}}} \operatorname{tr}(\mathbf{AX}) = \frac{\partial}{\partial \mathbf{X_{ij}}} (\sum_r \sum_s a_{rs}x_{sr}) =a_{ji} $

(2)In another section 'Conversion from differential to derivative form' it is shown the canonical form $dy = \operatorname{tr}(\mathbf{A}\,d\mathbf{X}) $ is equivalent to differential form$\frac{dy}{d\mathbf{X}} = \mathbf{A}$.

Can anyone help please to understand the this conversion? How this conversion is related to to formula (1)? I thought if they are related then the $\frac{dy}{d\mathbf{X}} = \mathbf{A}^\top$.

I found this proof from the following note.

enter image description here

but I don't understand what happen in the last part ( It is shown inside the red box). I assumed it will be $a_{ji}$. Why the derivative is written with respect to $x_{ji}$.

I would appreciate any insights on this.

Thank you.


Let $f:X\in M_n\rightarrow tr(AX)$. Your first equality concerns the gradient of $f$. The derivative of $f$ in $A$ is the following linear application:

$Df_A:H\in M_n\rightarrow tr(AH)\in \mathbb{R}$ (that is $Df_A=f$, because $f$ is linear!).

Note that $tr(AH)=<A^T,H>$ (the scalar product on $M_n$). The gradient is defined by duality: $<\nabla(f)(A),H>=Df_A(H)$, that is $\nabla(f)(A)=A^T$. In particular $\dfrac{\partial f}{\partial X_{i,j}}(A)=tr(AE_{i,j})=a_{j,i}=\nabla(f)(A)_{i,j}$.

Now, the matrix associated to the derivative $Df_A$ (or the differential) is a line with $n^2$ elements.

For example, if $n=2$: $tr(AX)=a_{1,1}x_{1,1}+a_{1,2} x_{2,1}+a_{2,1}x_{1,2}+a_{2,2}x_{2,2}$. If we stack the matrix $X$ column by column: $[x_{1,1},x_{2,1},x_{1,2},x_{2,2}]$, then we obtain for the derivative: $[a_{1,1},a_{1,2},a_{2,1},a_{2,2}]$; if we stack$^{-1}$ this last vector row by row, then we obtain $A$; yet, if we stack$^{-1}$ it column by column, then we obtain $A^T$. You choose the convention you want.

The author confuses with the case: $f:x\in\mathbb{R}^n\rightarrow \mathbb{R}$. The $1\times n$ matrix associated to $Df_A$ is $U=[\dfrac{\partial f}{\partial x_{1}},\cdots,\dfrac{\partial f_i}{\partial x_{n}}]$. The gradient of $f$ is the vector $V=[\dfrac{\partial f}{\partial x_{1}},\cdots,\dfrac{\partial f}{\partial x_{n}}]^T$ because (for $h\in\mathbb{R}^n$) $<V,h>=tr(V^Th)=V^Th$ and $Df(h)=Uh$, and then $V=U^T$.

EDIT. Answer to Crimson. The formula (1), given the gradient, is correct.

Rigorously, the formula (2) is not correct. Indeed $Df_A(H)=tr(AH)=tr((A\otimes I_n)(H))$ when we stack the matrices row by row (cf. https://en.wikipedia.org/wiki/Kronecker_product). Thus $Df_A$ is the composition $tr\circ (A\otimes I_n)$. If we stack column by column the variable and we stack$^{-1}$ row by row the image, then we find the matrix $A$. Clearly, it's not a fine dining!

  • $\begingroup$ Thank you for the answer. If you don't mind I have a question about the answer. I am not sure if I understood correctly. From the last part of your explanation I concluded that the formula given in (1) is column by column convention and the formula given in (2) is row by row convention. Is that correct? Thanks again. $\endgroup$ – Crimson Sep 25 '17 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.