Derivative w.r.t. the weight matrix in a linear layer? I'm deriving the back-propagation equations in a neural network. I have a single linear layer. I want to calculate the derivative of the loss function w.r.t. the weight matrix $W$. The Loss is some function on the output $Y$, $L(Y)$. $Y$ is given as $Y=XW^T$ where $Y, X, W$ are all matrices.
The answer is as below:
$$
\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Y} \frac{\partial Y}{\partial W} =?=(\frac{\partial L}{\partial Y})^T X
$$
I now want to derive that answer using index notation but am completely stuck. I first start by rewriting $Y$ for a single element:
$$
Y_{ij} = \sum^k X_{i,k}W_{j,k}
$$
But I don't really know how to complete the derivation using only index notation. And why does $\partial L / \partial Y$ become transposed, while it is before $\partial Y / \partial W$. I can derive that its necessary by investigating the shapes of the matrices, but that's not really sound reasoning imo.
 A: $
\def\l{\lambda}\def\o{{\tt1}}\def\p{\partial}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
$The Frobenius product is a concise notation for the trace
(or equivalently of hiding summations behind a product symbol)
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\
A:A &= \|A\|^2_F \\
}$$
The properties of the underlying trace function (or equivalently of the summation) allow the terms in such a product to be rearranged in many different ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
C:\LR{AB} &= \LR{CB^T}:A \\&= \LR{A^TC}:B \\
\\
}$$

Given a cost function $\l$, its gradient $G=\grad{\l}{Y}$ and the matrix relationship $Y=XW^T$ we can calculate the gradient with respect to the factors of $Y$ as follows.
Rewrite the gradient as a differential expression, perform a change variable from $Y\to W$, then rewrite the new differential expression in gradient form.
$$\eqalign{
d\l &= G:dY \\
  &= G:\LR{X\,dW^T} \\
  &= G^T:\LR{dW\,X^T} \\
  &= \LR{G^TX}:dW \\
\grad{\l}{W} &= {G^TX} \\
}$$
A similar calculation yields the gradient wrt $X$.
$$\eqalign{
d\l &= G:dY \\
  &= G:\LR{dX\,W^T} \\
  &= \LR{GW}:dX \\
\grad{\l}{X} &= {GW} \\
}$$
