4
$\begingroup$

I'm deriving the back-propagation equations in a neural network. I have a single linear layer. I want to calculate the derivative of the loss function w.r.t. the weight matrix $W$. The Loss is some function on the output $Y$, $L(Y)$. $Y$ is given as $Y=XW^T$ where $Y, X, W$ are all matrices.

The answer is as below:

$$ \frac{\partial L}{\partial W} = \frac{\partial L}{\partial Y} \frac{\partial Y}{\partial W} =?=(\frac{\partial L}{\partial Y})^T X $$

I now want to derive that answer using index notation but am completely stuck. I first start by rewriting $Y$ for a single element:

$$ Y_{ij} = \sum^k X_{i,k}W_{j,k} $$

But I don't really know how to complete the derivation using only index notation. And why does $\partial L / \partial Y$ become transposed, while it is before $\partial Y / \partial W$. I can derive that its necessary by investigating the shapes of the matrices, but that's not really sound reasoning imo.

$\endgroup$

1 Answer 1

1
$\begingroup$

$ \def\l{\lambda}\def\o{{\tt1}}\def\p{\partial} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} $The Frobenius product is a concise notation for the trace (or equivalently of hiding summations behind a product symbol) $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \|A\|^2_F \\ }$$ The properties of the underlying trace function (or equivalently of the summation) allow the terms in such a product to be rearranged in many different ways, e.g. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ C:\LR{AB} &= \LR{CB^T}:A \\&= \LR{A^TC}:B \\ \\ }$$


Given a cost function $\l$, its gradient $G=\grad{\l}{Y}$ and the matrix relationship $Y=XW^T$ we can calculate the gradient with respect to the factors of $Y$ as follows.

Rewrite the gradient as a differential expression, perform a change variable from $Y\to W$, then rewrite the new differential expression in gradient form. $$\eqalign{ d\l &= G:dY \\ &= G:\LR{X\,dW^T} \\ &= G^T:\LR{dW\,X^T} \\ &= \LR{G^TX}:dW \\ \grad{\l}{W} &= {G^TX} \\ }$$ A similar calculation yields the gradient wrt $X$. $$\eqalign{ d\l &= G:dY \\ &= G:\LR{dX\,W^T} \\ &= \LR{GW}:dX \\ \grad{\l}{X} &= {GW} \\ }$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .