# How to differentiate sum of matrix multiplication?

I try to solve the cost function minimization by differentiate matrix $A$. However, A is in the sum and at the middle of the product:

Solve $\frac{\delta F(A)}{\delta A} = 0$,

$0 = \frac{\delta }{\delta A} [\sum_{k} log (\sum_{c}S_{pc}A_{ck})H_{kn} ]$

where matrices $S$ is (p x c), $A$ is (c x k), and $H$ is (k x n)

I don't think I can apply just chain rule for $log(S_{pc}A_{ck}H_{kn}) = \frac{1}{S_{pc}A_{ck}H_{kn}} (S_{pc}A_{ck}H_{kn})'.$ How to deal with the sum before multiplication for matrix differentiation?

## 1 Answer

The gradient of a matrix-valued function with respect to a matrix argument will be a 4th order tensor. To avoid that difficulty, let's vectorize the matrix terms.

First, define some variables \eqalign{ B &= SAH \cr a &= {\rm vec}(A) \cr b &= {\rm vec}(B) &= (H^T\otimes S)\,a \cr y &= 1/b &\implies 1=y\circ b \cr Y &= {\rm Diag}(y) &\implies 1=Yb \cr f &= {\rm vec}(F) &= \log(b) \cr } Then find the differential and gradient of the function \eqalign{ df &= y\circ db = Y(H^T\otimes S)\,da \cr \frac{\partial f}{\partial a} &= Y(H^T\otimes S) \cr }

• I understand $vec(B)$ is Kronecker product but at the chain rule: $\partial f = y\circ db = Y(H^T\otimes S)\, \partial a$, how does $y$ turn into $Y$? I think $1$ at $1=y\circ b$ is vector of 1 (size (p x n) x 1) but $1$ at $1=Yb$ is diagonal matrix of 1 (size (p x n) x (p x n)). So $y \circ b$ is not equal to $Yb$? What am I wrong here? Why do we need $Y = Diag(y)$? – Jan Jan 17 '18 at 5:51
• The term $Yb$ is a matrix times a vector, which results in a vector and not a (diagonal) matrix. This is a standard technique for removing Hadamard products from an expression. – greg Jan 17 '18 at 20:05