With neural networks, back-propagation is an implementation of the chain rule. However, the chain rule is only applicable for differentiable functions. With non-differentiable functions, there is no chain rule that works in general. And so, it seems that back-propagation is invalid when we use a non-differentiable activation function (e.g. Relu).

The words that are stated around this seeming error is that "the chance of hitting a non-differentiable point during learning is practically 0". It's not clear to me, though, that landing on a non-differentiable point during learning is required in order to invalidate the chain rule.

Is there some reason why we should expect back-propagation to yield an estimate of the (sub)gradient? If not, why does training a neural network usually work?

  • $\begingroup$ Just a note, there is a SE site focused on machine learning: stats.stackexchange.com , just in case you won't get an answer here. Also somehow related (on yet another SE site that is currently in the beta): Differentiable activation function. DuttaA's answer seems to be especially interesting. Also look at quora.com/… , it mentions interesting concept called Subderivative (en.m.wikipedia.org/wiki/Subderivative). $\endgroup$
    – Sil
    Jul 1 '18 at 17:02
  • 1
    $\begingroup$ A NN with a smooth activating function like the logistic function, depends continuously on it's parameters. The back-propagation process appears when minimizing the data fitting error which is also a continuous function of it's parameters. Concluding, with a smooth activating function, the chain rule is quite operative and represents the chain rule. $\endgroup$
    – Cesareo
    Jul 1 '18 at 17:07
  • 2
    $\begingroup$ @Cesareo The point is that non-smooth activation functions are used, like ReLU. What happens then? $\endgroup$
    – rubik
    Jul 2 '18 at 6:40
  • 1
    $\begingroup$ @user3658307 Vandenberghe's 236c notes on gradient descent contain an example where gradient descent with exact line search fails to find a global minimizer for a convex but nondifferentiable function, despite the fact that the method never encounters a point where the objective function is nondifferentiable. See slide 1-5 here: seas.ucla.edu/~vandenbe/236C/lectures/gradient.pdf $\endgroup$
    – littleO
    Jul 4 '18 at 9:19
  • 1
    $\begingroup$ "It's not clear to me, though, that landing on a non-differentiable point during learning is required in order to invalidate the chain rule" The chain rule itself works fine as long as we have not landed on a non-differentiable point. If $f = g \circ h$ and $h$ is differentiable at $x$, and $g$ is differentiable at $h(x)$, then $f$ is guaranteed to be differentiable at $x$ and $f'(x) = g'(h(x)) h'(x)$. It seems to me that the real question is: is there any theoretical guarantee that gradient descent performs well provided that we avoid nondifferentiable points. (See previous comment.) $\endgroup$
    – littleO
    Jul 4 '18 at 9:31

The answer to this question might be more clear now with the following two papers:

  1. Kakade and Lee (2018) https://papers.nips.cc/paper/7943-provably-correct-automatic-sub-differentiation-for-qualified-programs.pdf

  2. Bolte and Pauwels (2019) https://arxiv.org/pdf/1909.10300.pdf

As you say, it is wrong to use the chain rule with ReLU activation functions. Evenmore the argument that "the output is differentiable almost everywhere implies that the classical chain rule of differentiation applies almost everywhere" is False. see Remark 12 in the second reference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.