Using Chain Rule in Matrix Differentiation I have the following parameters and their respective dimension:
$X:2\times 1$, $W_1:7\times2$, $W_2:1\times7$, $B_1: 7\times 1$ and $B_2:1\times1$, with the following formulation:
$Y=W_2H+B_2$ where $H=\verb+ReLU+(W_1X+B_1)$, the rectified linear unit applied element-wise ($\verb+ReLU+(x)=\max(0,x)$). I want to compute $$\frac{\partial Y}{\partial W_1},$$
by using the chain rule. Hence, I compute 
$$\frac{\partial Y}{\partial W_1}=\frac{\partial Y}{\partial H}\cdot\frac{\partial H}{\partial W_1}$$
which is equal to $$W_2\cdot\frac{\partial H}{\partial W_1}.$$
My problem is computing $\frac{\partial H}{\partial W_1}$. I take out $X^T$ from this, by using chain rule, but then it doesn't match the dimesnion for multiplication. How do we go about taking the derivative of $H$ w.r.t. $W_1$, which is a $7\times 2$ matrix? 
I was told that the final result has dimesnion $7\times2$, but no matter how I arrange things, I can't come up with the correct result. 
 A: Rather than the chain rule, let's tackle the problem using differentials.
Let's use the convention that an upppercase letter is a matrix, lowercase is a column vector, and a greek letter is a scalar. Now let's define some variables and their differentials
$$\eqalign{
 z &= W_1x+b_1  &\implies dz=dW_1\,x \cr
 h &= {\rm relu}(z) &\implies dh={\rm step}(z)\odot dz = s\odot dz \cr
}$$ where ${\rm step}(z)$ is the Heaviside step function. Both the relu and step functions are applied elementwise to their arguments.
Now take the differential and gradient of the function.
$$\eqalign{
 \phi &= w_2^Th + \beta_2 \cr
  &= w_2:h + \beta_2 \cr
\cr
d\phi
 &= w_2:dh \cr
 &= w_2:(s\odot dz) \cr
 &= (s\odot w_2):dz \cr
 &= (s\odot w_2):dW_1\,x \cr
 &= (s\odot w_2)x^T:dW_1 \cr
\cr
\frac{\partial\phi}{\partial W_1}
 &= (w_2\odot s)x^T \cr
 &= \Big(w_2\odot{\rm step}(W_1x+b_1)\Big)x^T \cr\cr
}$$
In the above, I used the notations
$$\eqalign{
 &A:B = {\rm tr}(A^TB) \cr
 &A\odot B \cr
}$$
for the trace/Frobenius and elementwise/Hadamard products, respectively.
A: Let us consider only $W_2W_1X$ since the rest is quite simple. You should understand better how derivatives work. The best is to take first a "directional derivative", in this case
$$
\lim_{h\to0}\frac1h[W_2(W_1+hV)X-W_2W_1X]=\frac1h[W_2(W_1+hV)X-W_2W_1X]=W_2VX.
$$
This means that the derivative with respect to $W_1$ is the linear transformation $T(V)=W_2VX$ that certainly can be identified with a $7\times2$ matrix since you need $14$ entries. But $T$ doesn't act by multiplication by that $7\times2$ matrix in an such identification.
PS: Rolled back to the original since @Friedrich Philipp didn't understand the answer and changed it!
