0
$\begingroup$

I am trying to understand how matrix derivatives work, focusing myself on the chain rule.

Consider $g(U): \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}$, and $U=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$

Then applying the chain rule I know that:

$\frac{\partial g(U)}{\partial X_{ij}}=\text{Tr}[(\frac{\partial g(U)}{\partial U})^T\frac{\partial U}{\partial X_{ij}}]$

However, what happens if $g(U): \mathbb{R}^{N\text{x}N} \rightarrow ^{N\text{x}N}$. I mean if I have to take the derivative of a matrix w.r.t a matrix. This could appear, for instance, if we have that $U=f(Z) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$ and $Z=f(X) : \mathbb{R}^{N\text{x}N} \rightarrow \mathbb{R}^{N\text{x}N}$, as on of the steps of the chain rule will involve the derivative of $U$ w.r.t $Z$

$\endgroup$
1
  • $\begingroup$ yes totally agree. The thing is that my previous question did not appeared in my profile so I though it was unpublished. That is why I posted it again. $\endgroup$
    – jdeJuan
    Oct 23 '19 at 8:09
1
$\begingroup$

Let's assume that $f$ can be expanded as a power series, i.e. $$\eqalign{ U &= f(X) = \sum_{k=0}^{\infty} \beta_kX^k \\ dU &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ }$$ You've told us nothing about the $g(U)$ function, but let's also assume you know how to calculate its gradient $$G = \frac{\partial g}{\partial U} \quad\implies dg = G:dU$$ where the colon denotes the trace/Frobenius product, i.e. $\;A:B={\rm Tr}(A^TB)$.

Combining these results yields. $$\eqalign{ dg &= G:\sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} X^{j}\,dX\,X^{k-j-1} \\ &= \sum_{k=0}^{\infty}\beta_k \sum_{j=0}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \,:\,dX \\ \frac{\partial g}{\partial X} &= \sum_{k=0}^{\infty}\beta_k \sum_{j=1}^{k-1} \Big[X^{k-j-1}\,G^T\,X^{j}\Big]^T \\ }$$ Thus one can calculate the desired gradient without calculating the 4th-order tensor $\,\frac{\partial U}{\partial X}$

$\endgroup$
6
  • $\begingroup$ yes the $g(U)$ function can be something like a trace. Thank's for the answer. $\endgroup$
    – jdeJuan
    Oct 23 '19 at 16:05
  • $\begingroup$ one more thing, the order of the tensor from $\frac{\partial U}{\partial X}$ is not 3, instead of 4?. By three I mean a 3 dimensional matrix. $\endgroup$
    – jdeJuan
    Oct 23 '19 at 16:07
  • $\begingroup$ Consider the component form $\frac{\partial U_{ij}}{\partial X_{pq}}$. It's called a 4th order tensor because there are 4 free indices $(i,j,p,q)$ each of which ranges from $1\ldots n$ in this case. $\endgroup$
    – greg
    Oct 23 '19 at 16:13
  • $\begingroup$ oh perfect, thank you. But when using a computer it can be stored in a 3D array, right? $\endgroup$
    – jdeJuan
    Oct 23 '19 at 16:43
  • $\begingroup$ I think you missed the point of my post. There is no need to compute any higher-order tensors. The LHS is $\frac{\partial g}{\partial X}$, which is a matrix, as are all the terms on the RHS (except the $\beta_k$ scalar coefficients). And for common functions, the gradient is often much simpler than the generic result given above. $\endgroup$
    – greg
    Oct 23 '19 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.