If $y = W^T x$, what is $\frac{\partial y}{\partial W}$? I would like to derive the derivative of a vector by matrix, i.e. $y = W^Tx$, where $W$ is a matrix, $x,y$ are vectors. What is $\frac{\partial y}{\partial W} = \frac{\partial W^T x}{\partial W}$?

Follow-up:
Define another function $z = a^T y = a^T W^Tx$, so that $z$ is a scalar. We know that $\frac{\partial z}{\partial W}$ is a matrix with the same of $W$. At the same time, by chain rule
\begin{equation}
\frac{\partial z}{\partial W} = \frac{\partial z}{\partial y} \cdot \frac{\partial y}{\partial W} 
\end{equation}
where $\frac{\partial z}{\partial y}$ is a $1\times N$ vector ($N$ is the dimension of $y$). So it seems the dimensions of matrices in left and right hand side don't match. Any explanations?
 A: Using indices we have $y_i =\sum_j (W^T)_{ij} x_j=\sum_j W_{ji}x_j$.
So using kronecker delta:
$$ \frac{\partial y_i}{\partial W_{kl}} = \sum_j \delta_{ji,kl} x_j = \delta_{il} x_k.$$
For your second part with $z= a^Ty=a^T W^T x=\sum_{kl} a_l W_{kl} x_k$:
$$ a_l x_k = \frac{\partial z}{\partial W_{kl}} = \sum_i \frac{\partial z}{\partial y_i} \frac{\partial y_i}{\partial W_{kl}} = \sum_i a_i \delta_{il} x_k = a_l x_k .$$
Hope it helps.
A: Consider the full matrix version of this problem, written in terms of the Frobenius (:) Inner Product 
$$\eqalign{
 Y & = W^TX \cr
 z &= A:Y = XA^T:W  \cr
}$$
The gradient of $z$ can be evaluated directly
$$\eqalign{
 dz &= XA^T:dW \cr
\frac{\partial z}{\partial W} &= XA^T \cr\cr
}$$
It can also be evaluated by the chain rule
$$\eqalign{
 dz &= A:dY \cr
\frac{\partial z}{\partial Y} &= A \cr\cr
 dY &= dW^TX &= {\mathbb E}X^T:{\mathbb B}:dW \cr
\frac{\partial Y}{\partial W} &= {\mathbb E}X^T:{\mathbb B} \cr
}$$
Taking the inner product of these 2 gradients (note that the second one is a 4th order tensor) yields
$$\eqalign{
\frac{\partial z}{\partial W} &= \frac{\partial z}{\partial Y} :
\frac{\partial Y}{\partial W} \cr
 &= A:{\mathbb E}X^T:{\mathbb B} \cr
 &= AX^T:{\mathbb B} \cr
 &= XA^T \cr
}$$
The 4th order isotropic tensors have components
$$\eqalign{
 {\mathbb E}_{ijkl} &= \delta_{ik}\,\delta_{jl} \cr
 {\mathbb B}_{ijkl} &= \delta_{il}\,\delta_{jk} \cr
}$$
and have the interesting properties when multiplied by a matrix $M$, using the Frobenius product
$$\eqalign{
 {\mathbb E}:M = M:{\mathbb E} &= M \cr
 {\mathbb B}:M = M:{\mathbb B} &= M^T \cr
}$$
Having derived this for the full matrix case, the result also holds when the matrices $(Y, X)$ are replaced by the vectors $(y, x)$.
A: Let $f : \mathbb R^{m \times n} \to \mathbb R^n$ be defined by
$$f (\mathrm X) := \mathrm X^T \mathrm a$$
The $i$-th entry of $f$ is the scalar
$$f_i (\mathrm X) = \mathrm e_i^T \mathrm X^T \mathrm a = \mathrm a^T \mathrm X \mathrm e_i = \mbox{tr} (\mathrm e_i \mathrm a^T \mathrm X) = \mbox{tr} ((\mathrm a \mathrm e_i^T)^T \mathrm X) = \langle \mathrm a \mathrm e_i^T, \mathrm X \rangle$$
Hence, the derivative of $f_i$ with respect to $\mathrm X$ is the $m \times n$ matrix
$$\partial_{\mathrm X} f_i (\mathrm X) = \mathrm a \mathrm e_i^T$$
