With neural networks, back-propagation is an implementation of the chain rule. However, the chain rule is only applicable for differentiable functions. With non-differentiable functions, there is no chain rule that works in general. And so, it seems that back-propagation is invalid when we use a non-differentiable activation function (e.g. Relu).

The words that are stated around this seeming error is that "the chance of hitting a non-differentiable point during learning is practically 0". It's not clear to me, though, that landing on a non-differentiable point during learning is required in order to invalidate the chain rule.

Is there some reason why we should expect back-propagation to yield an estimate of the (sub)gradient? If not, why does training a neural network usually work?

  • $\begingroup$ Just a note, there is a SE site focused on machine learning: stats.stackexchange.com , just in case you won't get an answer here. Also somehow related (on yet another SE site that is currently in the beta): Differentiable activation function. DuttaA's answer seems to be especially interesting. Also look at quora.com/… , it mentions interesting concept called Subderivative (en.m.wikipedia.org/wiki/Subderivative). $\endgroup$ – Sil Jul 1 '18 at 17:02
  • 1
    $\begingroup$ A NN with a smooth activating function like the logistic function, depends continuously on it's parameters. The back-propagation process appears when minimizing the data fitting error which is also a continuous function of it's parameters. Concluding, with a smooth activating function, the chain rule is quite operative and represents the chain rule. $\endgroup$ – Cesareo Jul 1 '18 at 17:07
  • 2
    $\begingroup$ @Cesareo The point is that non-smooth activation functions are used, like ReLU. What happens then? $\endgroup$ – rubik Jul 2 '18 at 6:40
  • 1
    $\begingroup$ @user3658307 Vandenberghe's 236c notes on gradient descent contain an example where gradient descent with exact line search fails to find a global minimizer for a convex but nondifferentiable function, despite the fact that the method never encounters a point where the objective function is nondifferentiable. See slide 1-5 here: seas.ucla.edu/~vandenbe/236C/lectures/gradient.pdf $\endgroup$ – littleO Jul 4 '18 at 9:19
  • 1
    $\begingroup$ "It's not clear to me, though, that landing on a non-differentiable point during learning is required in order to invalidate the chain rule" The chain rule itself works fine as long as we have not landed on a non-differentiable point. If $f = g \circ h$ and $h$ is differentiable at $x$, and $g$ is differentiable at $h(x)$, then $f$ is guaranteed to be differentiable at $x$ and $f'(x) = g'(h(x)) h'(x)$. It seems to me that the real question is: is there any theoretical guarantee that gradient descent performs well provided that we avoid nondifferentiable points. (See previous comment.) $\endgroup$ – littleO Jul 4 '18 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.