1
$\begingroup$

Suppose we need to differentiate a scalar which is the product of several matrix multiplications and Hadamard (elementwise) products between matrices. $$ Y= (A(B(XC)\circ D)\circ E)F$$ $$\frac{\partial Y}{\partial X}=?$$

Let the dimensions be A: (1*a), B: (a*b), X: (b*1), C(1*e), D(b* e), E(a*e), F(e*1)

So, Y is a scalar and we are differentiating it with respect vector X. Therefore, we expect the derivative to be a b*1 vector, like X.

i) First of all, unless we vectorise all matrices, I don't think we can rearrange the Hadamard product as matrix multiplication, which is quite inconvenient in this case.

I am trying to apply product rule and chain rule in order to figure it out but I am running into several problems.

ii) I am not sure how the chain rule can work in this case, because, when breaking down the function, we run into differentiation of Matrix over vector (e.g. $(B(XC)\circ D)$ is an a*e matrix)

iii) Moreover, I am not sure how the dimensions of the matrices can match after the differentiation (i.e. after $X$ is removed). Some suggest using the Kronecker Product, but I do not see how this can result into a b*1 vector in the end.

So, if someone can calculate the derivative here and show us how to get to a vector matching the dimensions of X, it will be very much appreciated.

$\endgroup$

1 Answer 1

4
$\begingroup$

First some notation. Denote the trace/Frobenius product with a colon, i.e. $$A:B = {\rm Tr}(A^TB)$$ a matrix with an uppercase letter, a vector with a lowercase letter, and a scalar with a Greek letter.

For typing convenience, define the column vectors $$\eqalign{ a &= A^T, \quad c &= C^T, \quad f &= F, \quad x &= X \\ }$$ and the matrices $$\eqalign{ H &= B^T\big(E\odot af^T\big), \quad K &= H\odot D \\ }$$ Rewrite the function in terms of these new variables. $$\eqalign{ \gamma &= a^T\big(B(xc^T\odot D)\odot E\big)f \\ &= a:\big(B(xc^T\odot D)\odot E\big)f \\ &= af^T:\big(B(xc^T\odot D)\odot E\big) \\ &= (E\odot af^T):B(xc^T\odot D) \\ &= H:(xc^T\odot D) \\ &= K:xc^T \\ &= Kc:x \\ }$$ Now it's a simple matter to find the differential and gradient. $$\eqalign{ d\gamma &= Kc:dx \\ \frac{\partial \gamma}{\partial x} &= Kc \\ }$$ NB:   The properties of the trace allow Frobenius products to be rearranged in a variety of ways. $$\eqalign{ A:B &= A^T:B^T \\ A:BC &= AC^T:B \;=\; B^TA:C \\ }$$ Also, Hadamard and Frobenius products commute with themselves and each other. $$\eqalign{ A:B &= B:A \\ A\odot B &= B\odot A \\ C:A\odot B &= C\odot A:B \\ }$$

Update

There was a question in the comments about the related vector-valued problem $$y = A\big(B(xc^T\odot D)\odot E\big)f$$ Even for this modified problem, the chain rule remains impractical. The real difficulty with both problems is the presence of the Hadamard products $-$ they make things awkward.

Nonetheless, here is how to calculate the gradient of the modified problem.

First, define some new variables. $$\eqalign{ C &= {\rm Diag}(c), \quad X = {\rm Diag}(x)\; \implies\;B(xc^T\odot D) = B(XDC) \\ E &= \sum_k \sigma_ku_kv_k^T \quad {\rm \{SVD\}} \\ W_k &= {\rm Diag}(\sigma_ku_k), \; V_k = {\rm Diag}(v_k) \implies E\odot Z = \sum_k W_k Z V_k \\ }$$ Then rewrite the function. $$\eqalign{ y &= A(E\odot BXDC)\,f \\ &= \sum_k A(W_kBXDCV_k)f \\ &= \sum_k {\rm vec}\Big(AW_kB\quad{\rm Diag}(x)\quad DCV_kf\Big) \\ &= \sum_k {\rm vec}\Big(\alpha_k\,{\rm Diag}(x)\,\beta_k\Big) \\ &= Jx\\ }$$ where this result provides a closed-form expression for the $J$-matrix. $$\eqalign{ J &= \sum_k (\beta_k^T\otimes {\tt 1})\odot({\tt 1}\otimes \alpha_k) \\ }$$ Having rewritten the problem in this form, the gradient (i.e. Jacobian) is trivial. $$\eqalign{ \frac{\partial y}{\partial x} &= J \\ }$$

$\endgroup$
4
  • $\begingroup$ Excellent solution! If a scalar is the product of matrices, you can always express it as the dot product of 2 vectors and make use of the lovely properties of traces as a result! However, going back to my question ii), I am wondering whether it is possible to get to the same answer using chain and product rule. The reason for insisting on this is that it would not be possible to work with traces (or only with traces) in cases where Y is a similar function, but not a scalar; say for instance that A is a q*a matrix in my example and as a result, Y is a vector and not a scalar. $\endgroup$ Commented Aug 30, 2019 at 12:28
  • $\begingroup$ I've updated the answer for a vector-valued function. You'll note that the chain rule was not used in the new solution either. That's because the chain rule is nearly useless in Matrix Calculus. Learn about differentials. They're much more practical and less error prone. $\endgroup$
    – greg
    Commented Aug 30, 2019 at 15:25
  • $\begingroup$ Could you please have a look at this question as well @greg? link $\endgroup$ Commented Sep 10, 2019 at 19:00
  • $\begingroup$ Can you also take a look at this one, which extends the problem to a second order vector? link I get the same result applying a solution like yours using traces and via product rule (which is applicable in this case I think). However, it makes no sense that I do not end up with a full rank coefficient. If you agree with my solution, please let me know in a comment and I will explain in a subsequent comment what the paradox is. Thanks @greg $\endgroup$ Commented Sep 17, 2019 at 12:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .