I have the following situation: I need to calculate the gradient of the Error of a CNN a few layers deep by hand. Starting with the Error function,

The $\operatorname{Error}[readoutX]= -\sum_i \sum_j {actualX_{ij} * \operatorname{log}(readoutX_{ij})}$

So, letting $actualX = \mathbf{a}$ and $readoutX = \mathbf{x}$, I need to take the gradient of the Error function.

The error function written out more fully, is:

$$ \begin{align} \operatorname[X] &= -\sum_i \sum_j a_{ij} * \operatorname{log}(x_{ij})\\ \mathbf{x} &= \operatorname{softmax}(\mathbf{B}\bullet \mathbf{A})\\ \mathbf{B} &= \mathbf{C} \bullet \mathbf{B}\\ \end{align} $$

$\text{ Where the actual dimensions of }\mathbf{A},\mathbf{B}, \text{ and }\mathbf{C} \text{ are } (1024,10), (1568,1024),(?,1568) \text{ respectively (}? = \text{number of images in batch}), \text{ and the softmax is defined as: }$ $$\sigma(\mathbf{x})_j = \frac{e^{x_j}}{\sum_{k=1}^{|\mathbf{x}|} e^{x_k}} \text{ for } j = 1,\ldots,|\mathbf{x}| $$

$\text{ with the partial derivative with respect to element }x_j:$

$$ \frac{\partial{\sigma(x)_i}}{\partial{x_j}} = \sigma(x)_i (\Delta_{ij} - \sigma(x)_j) $$

Next, I start to write the gradient:

$$ \nabla\operatorname{Error}[\mathbf{x}] = \sum_i \sum_j {- \frac{\partial{(a_{ij} * \operatorname{log}(x_{ij}))}}{\partial{x_{ij}}}} = \sum_i\sum_j -a_{ij}\frac{\partial{\operatorname{log}(x_{ij})}}{\partial{x_{ij}}} = \sum_i\sum_j -a_{ij}\frac{1}{x_{ij}}\frac{\partial{x_{ij}}}{\partial{u}} $$

And I am stuck at: $\frac{\partial{x_{ij}}}{\partial{u}}$.

I know that $\frac{\partial{x_{ij}}}{\partial{u}}$ should be a gradient, but I don't know how to implement it (I can just write the $\nabla$ in place of the partial and call it correct...but I have no idea if that is right...), and even after reading the Wikipedia article, I am not sure which chain rule applies here (or even if the first part of my gradient is correct). In the past, I've done this using matrix derivatives, but it seems like the summation breaks that down...

So, how do I write this first step...and transition to the next gradient (and then on to $\mathbf{C}$)...


Let $$\eqalign{ M &= A^TB^T\otimes I_c \cr c &= {\rm vec}(C) \cr z &= Mc \cr X &= {\rm softmax}(CBA) \cr x &= {\rm vec}(X) = {\rm softmax}(z) \cr y &= {\rm vec}(a) \cr\cr }$$ For our purposes, the most convenient form for differential of the softmax function is $$\eqalign{ dx = ({\rm Diag}(x) - xx^T)\,dz \cr\cr }$$ Write the error function in terms of the Frobenius(:) Inner Product, and find its differential $$\eqalign{ E &= -y:\log(x) \cr\cr dE &= -y:d\log(x) \cr &= -y:\frac{dx}{x} \cr &= -\frac{y}{x}:dx \cr &= -\frac{y}{x}:({\rm Diag}(x) - xx^T)\,dz \cr &= (xx^T-{\rm Diag}(x))\,\Big(\frac{y}{x}\Big) : dz \cr &= (xx^T-{\rm Diag}(x))\,\Big(\frac{y}{x}\Big) : M\,dc \cr &= M^T(xx^T-{\rm Diag}(x))\,\Big(\frac{y}{x}\Big) : dc \cr }$$ where $\frac{y}{x}$ represents element-wise (aka Hadamard) division of two vectors.

Since $dE=\big(\frac{\partial E}{\partial c}:dc\big),\,$ looking at that last line, the gradient must be $$\eqalign{ \frac{\partial E}{\partial c} &= M^T(xx^T-{\rm Diag}(x))\,\Big(\frac{y}{x}\Big) \cr }$$

  • $\begingroup$ wow <--- now I get it haha $\endgroup$
    – Chris
    Sep 19 '17 at 0:38
  • $\begingroup$ suppose my softmax is $a \times b$, $a < b$. I should just pad out the diagonal? $\endgroup$
    – Chris
    Sep 19 '17 at 0:58
  • 2
    $\begingroup$ This result can be reduced further since $$\eqalign{\frac{\partial E}{\partial c}&=M^T(xx^T-{\rm Diag}(x))\Big(\frac{y}{x}\Big)\cr&=M^T\Big(xx^T-{\rm Diag}(x)\Big){\rm Diag}(x)^{-1}y\cr&= M^T(x1^T-I)\,y\cr&=M^T(x-y)\cr}$$ $\endgroup$
    – lynn
    Nov 13 '17 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.