Chain Rule in Matrix Derivatives

I am trying to understand how matrix derivatives work, focusing myself on the chain rule.

Consider $$g(U): \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}$$, and $$U=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$$

Then applying the chain rule I know that:

$$\frac{\partial g(U)}{\partial X_{ij}}=\text{Tr}[(\frac{\partial g(U)}{\partial U})^T\frac{\partial U}{\partial X_{ij}}]$$

However, what happens if $$g(U): \mathbb{R}^{N\text{x}N} \rightarrow ^{N\text{x}N}$$. I mean if I have to take the derivative of a matrix w.r.t a matrix. This could appear, for instance, if we have that $$U=f(Z) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$$ and $$Z=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$$, as on of the steps of the chain rule will involve the derivative of $$U$$ w.r.t $$Z$$

• yes totally agree. The thing is that my previous question did not appeared in my profile so I though it was unpublished. That is why I posted it again. Oct 23 '19 at 8:09

Let's assume that $$f$$ can be expanded as a power series, i.e. \eqalign{ U &= f(X) = \sum_{k=0}^{\infty} \beta_kX^k \\ dU &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ } You've told us nothing about the $$g(U)$$ function, but let's also assume you know how to calculate its gradient $$G = \frac{\partial g}{\partial U} \quad\implies dg = G:dU$$ where the colon denotes the trace/Frobenius product, i.e. $$\;A:B={\rm Tr}(A^TB)$$.

Combining these results yields. \eqalign{ dg &= G:\sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \,:\,dX \\ \frac{\partial g}{\partial X} &= \sum_{k=0}^{\infty}\beta_k \sum_{j=1}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \\ } Thus one can calculate the desired gradient without calculating the 4th-order tensor $$\,\frac{\partial U}{\partial X}$$

• yes the $g(U)$ function can be something like a trace. Thank's for the answer. Oct 23 '19 at 16:05
• one more thing, the order of the tensor from $\frac{\partial U}{\partial X}$ is not 3, instead of 4?. By three I mean a 3 dimensional matrix. Oct 23 '19 at 16:07
• Consider the component form $\frac{\partial U_{ij}}{\partial X_{pq}}$. It's called a 4th order tensor because there are 4 free indices $(i,j,p,q)$ each of which ranges from $1\ldots n$ in this case.
– greg
Oct 23 '19 at 16:13
• oh perfect, thank you. But when using a computer it can be stored in a 3D array, right? Oct 23 '19 at 16:43
• I think you missed the point of my post. There is no need to compute any higher-order tensors. The LHS is $\frac{\partial g}{\partial X}$, which is a matrix, as are all the terms on the RHS (except the $\beta_k$ scalar coefficients). And for common functions, the gradient is often much simpler than the generic result given above.
– greg
Oct 23 '19 at 20:46