I'm kind of rusty in calculus.

Why is the ReLU function not differentiable at $f(0)$?

$$ f(x) = \begin{cases} 0 & \text{if $x \leq 0$} \\ x & \text{if $x > 0$}. \end{cases} $$

  • 1
    $\begingroup$ You should say “differentiable at $0$”, not “differentiable at $f(0)$”. (Not that it actually matters in this case, since $f(0)$ happens to be $0$, but anyway...) $\endgroup$ – Hans Lundmark Aug 8 '19 at 7:29

If you look at $x > 0$, or the righthand derivative, $$\frac{df}{dx} = \frac{d}{dx} x = 1$$ for all $x$.

If you look at $x \le 0$, or the lefthand derivative, $$\frac{df}{dx} = \frac{d}{dx} 0 = 0$$ for all $x$.

Since $x = 0$ is the "break" point, the lefthand and righthand derivatives are not the same, and thus, the derivative is not defined at $x = 0$.

| cite | improve this answer | |
  • 3
    $\begingroup$ The derivative can exist without being continuous. You have argued that $f$ can't be continuously differentiable, but how about differentiable? $\endgroup$ – Joonas Ilmavirta Aug 8 '19 at 7:21
  • $\begingroup$ I'm saying that the left and right hand derivatives are not the same at $x=0$. Therefore, the derivative does not exist. I'm not really saying anything about continuity. $\endgroup$ – Varun Vejalla Aug 8 '19 at 14:48
  • $\begingroup$ @JoonasIlmavirta Coming back to this answer after more than a year, could you expand on what you meant? I think I had misunderstood your comment before. $\endgroup$ – Varun Vejalla Aug 22 at 20:56
  • $\begingroup$ What you computed is $\lim_{x\to0\pm}f'(x)$, whereas the one-sided derivatives at zero are $\lim_{h\to0\pm}h^{-1}[f(h)-f(0)]$. These are not the same thing! A function can be everywhere non-differentiable but it can still have one or both one-sided derivatives at zero. You observed that $f$ is differentiable whenever $x\neq0$ and that the limits at $x=0$ disagree. This only shows that if $f$ is differentiable everywhere, it cannot be continuous, unless you are implicitly appealing to a theorem that bridges the gap for you. $\endgroup$ – Joonas Ilmavirta Aug 23 at 8:46
  • $\begingroup$ Ok, I think I understand now. I will update my answer accordingly. $\endgroup$ – Varun Vejalla Aug 23 at 17:07

Because it has a sharp corner at $0$, so it doesn't have a well defined tangent line; think about it, you can imagine many lines going through $(0, 0)$ that are tangent to the graph, so there are many possible tangent lines.

More formally, we have to investigate the limit

$$\lim_{h \to 0} \dfrac{f(0+h) - f(0)}{h}$$

This limit does not exist for the function, because if you let $h$ approach $0$ from the right, you get

$$\lim_{h \to 0^+} \dfrac {h-0}{h}=1.$$

While if you let $h$ approach $0$ from the left, that limit

$$\lim_{h \to 0^-} \dfrac {0-0}{h}=0.$$

Therefore the limit does not exist, so the function is not differentiable at $0$.

| cite | improve this answer | |
  • $\begingroup$ "a tangent line is one which touches the graph only at one point" isn't entirely true. Consider the tangent of $x^2(x-1)$ at $x = 0$. It intersects the graph at $x = 1$. $\endgroup$ – Arthur Aug 8 '19 at 6:55
  • $\begingroup$ I would not call that sharp corner or bend at $0$ a "cusp". A cusp is when the curve kind of turns 180 degrees so that the incoming tangent vector has the opposite direction of the outgoing tangent vector. The function $x\mapsto\sqrt{|x|}$ has a cusp. $\endgroup$ – Jeppe Stig Nielsen Aug 8 '19 at 7:27
  • $\begingroup$ @Arthur Yeah I'm aware, but I think for the purposes of the OP it's fine. But I'm not actually sure of the definition of tangent line myself; the best thing I could think of is: "L is tangent to $f$ at $x$ if there exists some neighborhood around $x$ in which $L$ is either completely below or completely above the graph of $f$." This definition seems satisfactory to me from a geometric point of view; but then we would have to give up the notion that "derivatives give tangent lines," because for example $x^2 \sin (\frac 1x).$ $\endgroup$ – Ovi Aug 8 '19 at 13:44
  • $\begingroup$ The tangent of $x^3$ at $x = 0$ then becomes problematic. I don't think you can get a proper definition of tangent without going into derivatives, or at least the same kinds of limits as the ones that define derivatives. $\endgroup$ – Arthur Aug 8 '19 at 13:47
  • $\begingroup$ @JeppeStigNielsen Oh thanks for the info, I wasn't aware; my highschool teacher told us all of these sharp corners are cusps. I have fixed the answer. $\endgroup$ – Ovi Aug 8 '19 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.