I try to solve the cost function minimization by differentiate matrix $A$. However, A is in the sum and at the middle of the product:

Solve $\frac{\delta F(A)}{\delta A} = 0$,

$0 = \frac{\delta }{\delta A} [\sum_{k} log (\sum_{c}S_{pc}A_{ck})H_{kn} ]$

where matrices $S$ is (p x c), $A$ is (c x k), and $H$ is (k x n)

I don't think I can apply just chain rule for $log(S_{pc}A_{ck}H_{kn}) = \frac{1}{S_{pc}A_{ck}H_{kn}} (S_{pc}A_{ck}H_{kn})'. $ How to deal with the sum before multiplication for matrix differentiation?


The gradient of a matrix-valued function with respect to a matrix argument will be a 4th order tensor. To avoid that difficulty, let's vectorize the matrix terms.

First, define some variables $$\eqalign{ B &= SAH \cr a &= {\rm vec}(A) \cr b &= {\rm vec}(B) &= (H^T\otimes S)\,a \cr y &= 1/b &\implies 1=y\circ b \cr Y &= {\rm Diag}(y) &\implies 1=Yb \cr f &= {\rm vec}(F) &= \log(b) \cr }$$ Then find the differential and gradient of the function $$\eqalign{ df &= y\circ db = Y(H^T\otimes S)\,da \cr \frac{\partial f}{\partial a} &= Y(H^T\otimes S) \cr }$$

  • $\begingroup$ I understand $vec(B)$ is Kronecker product but at the chain rule: $\partial f = y\circ db = Y(H^T\otimes S)\, \partial a$, how does $y$ turn into $Y$? I think $1$ at $1=y\circ b$ is vector of 1 (size (p x n) x 1) but $1$ at $1=Yb$ is diagonal matrix of 1 (size (p x n) x (p x n)). So $y \circ b$ is not equal to $Yb$? What am I wrong here? Why do we need $Y = Diag(y)$? $\endgroup$ – Jan Jan 17 '18 at 5:51
  • $\begingroup$ The term $Yb$ is a matrix times a vector, which results in a vector and not a (diagonal) matrix. This is a standard technique for removing Hadamard products from an expression. $\endgroup$ – greg Jan 17 '18 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.