# Simple matrix derivative

I always have some trouble doing matrix derivative, such as the following $$\frac{\partial(W^TX^T-Y^T)(XW-Y)}{\partial W} =\frac{\partial(W^TX^TXW - W^TX^TY - Y^TXW)}{\partial W}$$ After that, I do not know how to calculate matrix derivative when some term involves matrix transpose or matrix inverse. Can someone help me, and provide some rules regarding matrix derivatives, especially when the terms involves matrix transpose and matrix inverse? Thank you very much!

## 1 Answer

Take $X$ and $Y$ to be constant matrices, and define $f(W)\stackrel{\text{def}}{=}(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(XW-Y)$. Then by the distributive property of matrix multiplication and linearity of matrix transposition, $$\begin{split} f(W+h\Delta W)&=((W+h\Delta W)^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(X(W+h\Delta W)-Y)\\ &=(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})(XW-Y)\\ &\quad+h(\Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W)\\ &\quad + h^2\Delta W^{\mathsf{T}}X^{\mathsf{T}}X\Delta W \\ \therefore f(W+h\Delta W)&=f(W)+h\langle\nabla f(W),\Delta W\rangle+o(h) \end{split}$$ where the linear map $\nabla f(W)$ is defined by $$\langle\nabla f(W),\Delta W\rangle\stackrel{\text{def}}{=}\Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W\text{.}$$

By definition, $\nabla f(W)$ is the Gâteaux derivative of $f$ at $W$—for a change $\Delta W$ in $W$, it gives the "first-order" change in $f(W)$.

• The answer is $2X^TXW - 2X^TY$, but I just don't know how it is derived. – Steve Yang Mar 7 '18 at 20:34
• The derivative is the linear map $\langle \nabla f(W),\Delta W\rangle = \Delta W^{\mathsf{T}}X^{\mathsf{T}}(XW-Y)+(W^{\mathsf{T}}X^{\mathsf{T}}-Y^{\mathsf{T}})X\Delta W$—you can't simplify further unless you know the terms in this expression are symmetric. – K B Dave Mar 7 '18 at 20:46
• To calculate it, you do what I did: expand $f(W+h\Delta W)$ through first order in $h$. Then the coefficient of $h$ is linear in $\Delta W$, and this linear map is the derivative. – K B Dave Mar 7 '18 at 20:47
• what if X is a m*n matrix, W is a n*1 vector, and Y is a n*1 vector? – Steve Yang Mar 7 '18 at 21:10
• Everything already said still applies. But then $W$ is a vector and $f$ is a scalar, so this is a duplicate of math.stackexchange.com/questions/2594184/… – K B Dave Mar 7 '18 at 21:21