I have the following functions: $$\theta=Uh+b_2 \text{ where } U\in R^{m \times n},h\in R^{n\times1}$$ $$h=Wx+b_1 \text{ where } W\in R^{n\times k},x\in R^{k\times 1}$$ and when I calculate this derivative: $$\frac{\partial\theta}{\partial W}=\frac{\partial\theta}{\partial h}{\frac{\partial h}{\partial W}}=Ux$$ but it doesn't make sense because U is m,n and x is k,1. What am i doing wrong here ? and if you could please explain in general how to use the chain rule with these cases (I mean when i derive by matrix not scalar).

  • $\begingroup$ What is $R^{m,n}$? The space of $m\times n$ matrices with entries in $R$? $\endgroup$
    – MPW
    May 11, 2016 at 13:08
  • $\begingroup$ yes, I will fix it. $\endgroup$
    – Sefi
    May 11, 2016 at 13:10

1 Answer 1


Ok here goes nothing, haven't done this in a little bit. Hopefully I can still math.

Since you are taking the derivative with respect to the matrix $W$, for $h$ you will only care about the $Wh$ term. $Wh$ is an $n$x$1$ matrix and the $i^{th}$ component is $W_i^T$ $\cdot$ $x$. Now turning to $\theta$, we see that we really only care about $Uh$. Lets find the $l^{th}$ component of that. We it will be $\sum_{i=1}^nU_{li}(W_i^T$ $\cdot x)$ which looks like $U_{l1}(W_{11}x_1 + ...+W_{1K}x_k $) + ... +$U_{li}(W_{i1}x_1 + ...+W_{ik}x_k $)+...+$U_{ln}(W_{n1}x_1 + ...+W_{nK}x_k $) expanded. So you can take the term by term derivative with respect to $W$, ($\partial\theta_l/\partial W_{ij} = something$) and it is pretty straight forward to see that from the expansion that $\partial\theta_l/\partial W_{ij} = U_{li}x_j$ with $i\in[1,n]$ $j\in[1,k]$ and $l\in[1,m]$ so you we not off in your intuition. You just need to remember that derivative of matrix with respect to matrix is a tensor. And I think it is a $k$x$n$ matrix whose entries are $n$x$1$ vectors

$\partial\theta_l/\partial W_{11} = U_{l1}x_1$
$\partial\theta_l/\partial W_{1k} = U_{l1}x_k$
$\partial\theta_l/\partial W_{n1} = U_{ln}x_1$
$\partial\theta_l/\partial W_{nk} = U_{ln}x_k$


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .