I am trying to understand how matrix derivatives work, focusing myself on the chain rule.

Consider $g(U): \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}$, and $U=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$

Then applying the chain rule I know that:

$\frac{\partial g(U)}{\partial X_{ij}}=\text{Tr}[(\frac{\partial g(U)}{\partial U})^T\frac{\partial U}{\partial X_{ij}}]$

However, what happens if $g(U): \mathbb{R}^{N\text{x}N} \rightarrow ^{N\text{x}N}$. I mean if I have to take the derivative of a matrix w.r.t a matrix. This could appear, for instance, if we have that $U=f(Z) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$ and $Z=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$, as on of the steps of the chain rule will involve the derivative of $U$ w.r.t $Z$

  • $\begingroup$ yes totally agree. The thing is that my previous question did not appeared in my profile so I though it was unpublished. That is why I posted it again. $\endgroup$
    – jdeJuan
    Oct 23, 2019 at 8:09

1 Answer 1


Let's assume that $f$ can be expanded as a power series, i.e. $$\eqalign{ U &= f(X) = \sum_{k=0}^{\infty} \beta_kX^k \\ dU &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ }$$ You've told us nothing about the $g(U)$ function, but let's also assume you know how to calculate its gradient $$G = \frac{\partial g}{\partial U} \quad\implies dg = G:dU$$ where the colon denotes the trace/Frobenius product, i.e. $\;A:B={\rm Tr}(A^TB)$.

Combining these results yields. $$\eqalign{ dg &= G:\sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \,:\,dX \\ \frac{\partial g}{\partial X} &= \sum_{k=0}^{\infty}\beta_k \sum_{j=1}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \\ }$$ Thus one can calculate the desired gradient without calculating the 4th-order tensor $\,\frac{\partial U}{\partial X}$

  • $\begingroup$ yes the $g(U)$ function can be something like a trace. Thank's for the answer. $\endgroup$
    – jdeJuan
    Oct 23, 2019 at 16:05
  • $\begingroup$ one more thing, the order of the tensor from $\frac{\partial U}{\partial X}$ is not 3, instead of 4?. By three I mean a 3 dimensional matrix. $\endgroup$
    – jdeJuan
    Oct 23, 2019 at 16:07
  • $\begingroup$ Consider the component form $\frac{\partial U_{ij}}{\partial X_{pq}}$. It's called a 4th order tensor because there are 4 free indices $(i,j,p,q)$ each of which ranges from $1\ldots n$ in this case. $\endgroup$
    – greg
    Oct 23, 2019 at 16:13
  • $\begingroup$ oh perfect, thank you. But when using a computer it can be stored in a 3D array, right? $\endgroup$
    – jdeJuan
    Oct 23, 2019 at 16:43
  • 1
    $\begingroup$ I think you missed the point of my post. There is no need to compute any higher-order tensors. The LHS is $\frac{\partial g}{\partial X}$, which is a matrix, as are all the terms on the RHS (except the $\beta_k$ scalar coefficients). And for common functions, the gradient is often much simpler than the generic result given above. $\endgroup$
    – greg
    Oct 23, 2019 at 20:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .