I always have some trouble doing matrix derivative, such as the following $$\frac{\partial(W^TX^T-Y^T)(XW-Y)}{\partial W} =\frac{\partial(W^TX^TXW - W^TX^TY - Y^TXW)}{\partial W} $$ After that, I do not know how to calculate matrix derivative when some term involves matrix transpose or matrix inverse. Can someone help me, and provide some rules regarding matrix derivatives, especially when the terms involves matrix transpose and matrix inverse? Thank you very much!


Take $X$ and $Y$ to be constant matrices, and define $f(W)\stackrel{\text{def}}{=}(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(XW-Y)$. Then by the distributive property of matrix multiplication and linearity of matrix transposition, $$\begin{split} f(W+h\Delta W)&=((W+h\Delta W)^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(X(W+h\Delta W)-Y)\\ &=(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(XW-Y)\\ &\quad+h(\Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W)\\ &\quad + h^2\Delta W^{\mathsf{T}}X^{\mathsf{T}}X\Delta W \\ \therefore f(W+h\Delta W)&=f(W)+h\langle\nabla f(W),\Delta W\rangle+o(h) \end{split}$$ where the linear map $\nabla f(W)$ is defined by $$\langle\nabla f(W),\Delta W\rangle\stackrel{\text{def}}{=}\Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W\text{.}$$

By definition, $\nabla f(W)$ is the Gâteaux derivative of $f$ at $W$—for a change $\Delta W$ in $W$, it gives the "first-order" change in $f(W)$.

  • $\begingroup$ The answer is $2X^TXW - 2X^TY$, but I just don't know how it is derived. $\endgroup$ – Steve Yang Mar 7 '18 at 20:34
  • $\begingroup$ The derivative is the linear map $\langle \nabla f(W),\Delta W\rangle = \Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W$—you can't simplify further unless you know the terms in this expression are symmetric. $\endgroup$ – K B Dave Mar 7 '18 at 20:46
  • $\begingroup$ To calculate it, you do what I did: expand $f(W+h\Delta W)$ through first order in $h$. Then the coefficient of $h$ is linear in $\Delta W$, and this linear map is the derivative. $\endgroup$ – K B Dave Mar 7 '18 at 20:47
  • $\begingroup$ what if X is a m*n matrix, W is a n*1 vector, and Y is a n*1 vector? $\endgroup$ – Steve Yang Mar 7 '18 at 21:10
  • $\begingroup$ Everything already said still applies. But then $W$ is a vector and $f$ is a scalar, so this is a duplicate of math.stackexchange.com/questions/2594184/… $\endgroup$ – K B Dave Mar 7 '18 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.