Vector by Matrix derivative

Question

According to wikipedia, there is no widely accepted definition of a Vector by Matrix derivative. I have a need of such a notion.

For matrix w, and vector h. $$\mathbf{y=w \;h} $$

$$ \begin{bmatrix}y_{0}\\ y_{1}\\ \vdots\\ y_{n} \end{bmatrix}=\begin{bmatrix}w_{0,0} & w_{0,1} & \cdots & w_{0,k}\\ w_{0,0} & w_{1,1} & \cdots & w_{1,k}\\ \vdots & \cdots & \cdots & \vdots\\ w_{n,0} & \cdots & \cdots & w_{n,k} \end{bmatrix}\begin{bmatrix}h_{0}\\ h_{1}\\ \vdots\\ h_{k} \end{bmatrix}=\begin{bmatrix}h_{0}w_{00}+h_{1}w_{01}...+h_{k}w_{0k}\\ h_{0}w_{10}+h_{1}w_{11}...+h_{k}w_{1k}\\ \vdots\\ h_{0}w_{n0}+h_{1}w_{n1}...+h_{k}w_{nk} \end{bmatrix} $$

It can be seen with a little thought that $$\frac{d\mathbf{y}}{d\mathbf{h}}=\mathbf{w}$$, which is to say w is a jacobian.

However taking the element wise derivative of y with respect to each element of w is going to give me a number of sparse vectors, equal to the number of elements of w. I guess this could be groups into a 4d tensor, but i know nothing of them. Inparticular this looks like it is not going to work well with the chain-rule, which is essential to the larger problem i am trying to solve.

$$ \begin{array}{cc} \frac{d\mathbf{y}}{dw_{00}}=\begin{bmatrix}h_{0}\\ 0\\ \vdots\\ 0 \end{bmatrix} & \frac{d\mathbf{y}}{dw_{10}}=\begin{bmatrix}0\\ h_{0}\\ \vdots\\ 0 \end{bmatrix}\\ \frac{d\mathbf{y}}{dw_{101}}=\begin{bmatrix}h_{1}\\ 0\\ \vdots\\ 0 \end{bmatrix} & \frac{d\mathbf{y}}{dw_{11}}=\begin{bmatrix}0\\ h_{1}\\ \vdots\\ 0 \end{bmatrix} \end{array} $$

Unless I can find a good notion of this derivative, I am going to have to stop thinking about the problem in terms of matrix products, and instead think of it a very large number of equations based around the members of the the matrix. Thinking about it this way loses me abstraction, and if i am not clever enough it will increase my algorithms exectution time when it comes time to run this though BLAS Software (Basic Linear Algabra System, is the special software that access the functionality of the CPU that can do matrix math quickly)

I suspect thinking of it as elementwise equations is going to be my oldy way forward, most texts i've seen seem to be refering to these relationships, with definations like $$y_{i}=\sum_{0\le j\le k}h_{j}w_{i,j} $$

So my question: Is there a good notion of a vector by matrix derivative?

For reference I am attempting to rederive the backproperation learning algorithm for neural nets of arbitrary number of layers

Answer 1

Since $y = W\cdot h$, you can consider $h$ to be the independent variable and hold $W$ constant. This gives you $$dy = W\cdot dh$$ and as you observed, we can immediately identify the derivative as $$\frac{\partial{y}}{\partial{h}} = W$$

Now let's hold $h$ constant and vary $W$: $$\eqalign{ dy &= dW\cdot h \cr &= I\cdot dW\cdot h \cr &= Ih : dW \cr &\equiv \beta : dW \cr }$$ Once again, we can immediately identify the derivative as $$\frac{\partial{y}}{\partial{W}} = \beta$$ But this time $\beta$ is a $3^{rd}$ order tensor with components $$\beta_{ijk} = \delta_{ij} h_k $$