12
$\begingroup$

let $W$ be a $n\times m$ matrix and $\textbf{x}$ be a $m\times1$ vector. How do we calculate the following then?

$$\frac{dW\textbf{x}}{dW}$$

Thanks in advance.

$\endgroup$
  • 2
    $\begingroup$ That totally depends on definition being used. $\endgroup$ – Arjang Jan 22 '16 at 3:49
  • $\begingroup$ Also, I like your thinking, just make it explicit as what the definition should be. $\endgroup$ – Arjang Jan 22 '16 at 3:50
  • 2
    $\begingroup$ The wiki had "?" for this type of Matrix differentiation. "Result of differentiating various kinds of aggregates with other kinds of aggregates" in en.wikipedia.org/wiki/Matrix_calculus#Other_matrix_derivatives. I encountered this in the context of neural networks and not sure either how it's defined. $\endgroup$ – arindam mitra Jan 22 '16 at 4:00
-6
$\begingroup$

For independent case:

If $\mathbf{x}$ is independent of $W$, this problem can be calculated as follows.

$$\cfrac{\partial W\mathbf{x}}{\partial W}= \cfrac{\partial}{\partial W} \begin{bmatrix} w_{11} & w_{12} & \cdots & w_{1m} \\ w_{21} & w_{22} & \cdots & w_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ w_{n1} & w_{n2} & \cdots & w_{nm} \end{bmatrix} \mathbf{x} $$

$$ = \begin{bmatrix} \cfrac{\partial w_{11}}{\partial w_{11}} & \cfrac{\partial w_{12}}{\partial w_{12}} & \cdots & \cfrac{\partial w_{1m}}{\partial w_{1m}} \\ \cfrac{\partial w_{21}}{\partial w_{21}} & \cfrac{\partial w_{22}}{\partial w_{22}} & \cdots & \cfrac{\partial w_{2m}}{\partial w_{2m}} \\ \vdots & \vdots & \ddots & \vdots \\ \cfrac{\partial w_{n1}}{\partial w_{n1}} & \cfrac{\partial w_{n2}}{\partial w_{n2}} & \cdots & \cfrac{\partial w_{nm}}{\partial w_{nm}} \end{bmatrix} \mathbf{x} $$

Therefore, all elements are $1$. Eventually, the result is below.

$$ \cfrac{\partial W\mathbf{x}}{\partial W}= (\mathbf{x}^{\text{T}}\mathbf{1_{m}}) \mathbf{1_{n}} $$

Then $\mathbf{1_{k}} \in \mathbf{R}^{k}$ is

$$\mathbf{1_{k}}=[1 \ 1 \ \cdots 1]^{\text{T}}$$

For dependent case:

If $\mathbf{x}$ is dependent of $W$, it is more difficult than independent case. Likewise,

$$ \cfrac{\partial W\mathbf{x}}{\partial W}= (\mathbf{x}^{\text{T}}\mathbf{1_{m}}) \mathbf{1_{n}} + W \cfrac{\partial F(W) }{ \partial W}\mathbf{x}_{0} $$

Then, $\mathbf{x}$ can be replaced as follows

$$\mathbf{x}=F(W)\mathbf{x}_{0}$$

where, $F(W) \in \mathbf{R}^{m \times n}$ is a matrix function, for which parameters are $W$, and $\mathbf{x}_{0} \in \mathbf{R}^{m}$ is independent of $W$.

$\endgroup$
  • $\begingroup$ Hi, can you please point me to a book where I can learn more about derivatives w.r.t matrix? $\endgroup$ – arindam mitra Feb 1 '16 at 20:12
  • 11
    $\begingroup$ This may be the accepted answer, but it's just plain wrong. The problem is that the gradient $$\frac{\partial W}{\partial W} \ne 1_n1_m^T$$ as asserted. It is instead a 4th order tensor which can be written in index notation as $$\frac{\partial W_{ij}}{\partial W_{kl}}=\delta_{ik}\,\delta_{jl}$$ $\endgroup$ – greg Nov 6 '17 at 1:52
  • 1
    $\begingroup$ I fully agree with @greg. This answer is simply plain wrong... I am a bit scared to see that it has 4 upvotes... $\endgroup$ – Surb Jan 15 '18 at 13:50
18
$\begingroup$

The quantity in question is a $3^{rd}$ order tensor.

One approach is to use index notation $$\eqalign{ f_i &= W_{ij} x_j \cr\cr \frac{\partial f_i}{\partial W_{mn}} &= \frac{\partial W_{ij}}{\partial W_{mn}} \,x_j \cr &= \delta_{im}\delta_{jn} \,x_j \cr &= \delta_{im}\,x_n \cr }$$ Another approach is vectorization $$\eqalign{ f &= W\,x \cr &= I\,W\,x \cr &= (x^T\otimes I)\,{\rm vec}(W) \cr &= (x^T\otimes I)\,w \cr\cr \frac{\partial f}{\partial w} &= (x^T\otimes I) \cr }$$

$\endgroup$
  • $\begingroup$ Are you missing any $\sum_{}{}$ symbol? $\endgroup$ – arindam mitra Jan 22 '16 at 20:40
  • 2
    $\begingroup$ @arindammitra I guess lynn uses some Einstein summation notaion in the exrpression of $f_i$ $\endgroup$ – Surb Jan 15 '18 at 13:49
  • $\begingroup$ I don't think the rewriting of f is correct. At least it doesn't produce the same result when I try on a small example. $\endgroup$ – Sandi Nov 16 '18 at 12:44
  • 1
    $\begingroup$ I think the order of your $\mathbf{I}$ and $\mathbf{x}^T$ should be changed. $\endgroup$ – Sandi Nov 16 '18 at 12:52
  • 4
    $\begingroup$ @Sandi Your edits to lynn's answer are wrong. The correct vectorization formula is $${\rm vec}(IWx)=(x^T\otimes I){\rm vec}(W)$$ Please read the Wikipedia entry. This question must be cursed. The accepted answer is (still) wrong, and (now) lynn's answer has been corrupted. $\endgroup$ – greg Dec 21 '18 at 4:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.