If $A(x)$ was a row vector and $B(x)$ was a column vector, below is true $$ \frac{d(A(x)B(x))}{dx}=\frac{dA(x)}{dx}B(x)+\frac{dB^t(x)}{dx}A(x)^t. $$ If you take derivative of $d_1\times d_2$ matrix w.r.t. a length $p$ column vector you get $p\times d_1\times d_2$ tensor. If you take derivative of $d_1\times d_2$ matrix w.r.t. a length $p$ row vector you get $d_1\times d_2\times p$ tensor. Let's say $A(x)$ is $d_1\times d_2$, $B(x)$ is $d_2\times d_3$. Then dimensions are $$ \frac{d(A(x)B(x))}{dx}\qquad p\times d_1\times d_3\\ \frac{dA(x)}{dx}B(x)\qquad (p\times d_1\times d_2)\cdot (d_2\times d_3)\\ \frac{dB^t(x)}{dx}A(x)^t\qquad (p\times d_3\times d_2)\cdot (d_2\times d_1). $$ The last term with dimension $p\times d_3\times d_1$ is not what we want. Nor is it a transpose of what we want. Furthermore, if you take a derivative on $B(x)$ when you look at the dimension, $p$ always has to be next to $d_2$ or $d_3$ and can never be next to $d_1$. So it seems this issue is unresolvable. Any idea how to circumvent this?

  • $\begingroup$ Why do you transpose the second term? $(AB)'=A'B+AB'$. $\endgroup$ – Dog_69 Feb 23 '19 at 17:44
  • $\begingroup$ Because I want the derivative of an inner product w.r.t a column vector to be a column vector. $\endgroup$ – ztyh Feb 23 '19 at 17:57
  • $\begingroup$ Two things: the product of matrices is a representation for the composition of operators. So, as you have written, the number of columns of the first must be equal to the number of rows of the second. However, since they depend of the same parameter/variable, $x$, the must be defined on the same space. This fact means that $d_1=d_3$, and your problem is solved. $\endgroup$ – Dog_69 Feb 23 '19 at 20:06
  • $\begingroup$ So your second sentence says $d_2=d_2$, which is indeed true. I don't really follow your third sentence. I really just meant that $A$ and $B$ as matrices. They take different values according to $x$. I agree there will be no issue if $d_1=d_3$. $\endgroup$ – ztyh Feb 23 '19 at 21:41
  • $\begingroup$ What I said in that comment is that if they depend on $x$, is because they are defined over the same space, and thus $d_1$ and $d_3$ must agree. But that's false. It implies that $d_1=d_2$, but $d_3$ is free $\endgroup$ – Dog_69 Feb 23 '19 at 21:43

Index notation illustrates the problem quite well.

Denoting $\frac{\partial}{\partial x_n}$ by $\partial_n$, we have $$\eqalign{ C_{ik} &= A_{ij}B_{jk} \cr \partial_nC_{ik} &= \big(\partial_n A_{ij}\big)B_{jk} + A_{ij}\big({\partial_n B_{jk}}\big) \cr }$$ The first term is fine for the contraction over the $j$-index, but in the second term the index is sandwiched between the $n$ and $k$ indices. Since the term is a third-order tensor, there is no way to fix it. This is unlike the following case.

In the case where $(A,B)$ are vectors, simply omit the $(i,k)$ indices to obtain $$\eqalign{ C &= A_{j}B_{j} \cr \partial_nC &= \big(\partial_n A_{j}\big)B_{j} + A_{j}\big({\partial_n B_{j}}\big) \cr }$$ This expression can be fixed by transposing the second term.

Generalizing in the other direction, if $X$ is a matrix, $\frac{\partial}{\partial X_{nm}}=\partial_{nm}$, and appending the $m$-index results in $$\eqalign{ \partial_{nm}C_{ik} &= \big(\partial_{nm} A_{ij}\big)B_{jk} + A_{ij}\big({\partial_{nm} B_{jk}}\big) \cr }$$ Now the gradients in parentheses are fourth-order tensors, and once again there is no simple operation that will re-order the indices.

The simplest way to avoid the problem is to use differentials instead of gradients. $$\eqalign{ C &= A\star B \cr dC &= dA\star B + A\star dB \cr }$$ where $(A,B,C)$ can be scalars, vectors, or tensors, and $(\star)$ can be any kind of product (Kronecker, Hadamard, Frobenius, tensor, matrix) which is compatible with their given dimensions.

| cite | improve this answer | |
  • $\begingroup$ Doesn't the differential ignore $x$ now? It might be useful if you were studying the relationship between $C$ and $(A,B)$, but I am trying to study relationship between $C$ and $x$. As $(\partial A/\partial x,\partial B/\partial x)$ are the connecting pieces there is no way around them? Also is it that we just need to generalize the transpose? i.e. some more complicated switching of the dimensions? At least when I code it so that the operations conform, the result seems reasonable. $\endgroup$ – ztyh Feb 23 '19 at 22:07
  • 2
    $\begingroup$ Yes, you can do that. The results in index notation map directly to code. But your apparent goal was to write the results using standard matrix notation. And that's where you'll get stuck. You can always invent your own notation for such operations, but it won't be standard, so no one else will understand it without an explanation. $\endgroup$ – greg Feb 23 '19 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.