# Why ReLU function is not differentiable at 0?

I'm kind of rusty in calculus.

Why is the ReLU function not differentiable at $$f(0)$$?

$$f(x) = \begin{cases} 0 & \text{if x \leq 0} \\ x & \text{if x > 0}. \end{cases}$$

• You should say “differentiable at $0$”, not “differentiable at $f(0)$”. (Not that it actually matters in this case, since $f(0)$ happens to be $0$, but anyway...) – Hans Lundmark Aug 8 '19 at 7:29

If you look at $$x > 0$$, or the righthand derivative, $$\frac{df}{dx} = \frac{d}{dx} x = 1$$ for all $$x$$.

If you look at $$x \le 0$$, or the lefthand derivative, $$\frac{df}{dx} = \frac{d}{dx} 0 = 0$$ for all $$x$$.

Since $$x = 0$$ is the "break" point, the lefthand and righthand derivatives are not the same, and thus, the derivative is not defined at $$x = 0$$.

• The derivative can exist without being continuous. You have argued that $f$ can't be continuously differentiable, but how about differentiable? – Joonas Ilmavirta Aug 8 '19 at 7:21
• I'm saying that the left and right hand derivatives are not the same at $x=0$. Therefore, the derivative does not exist. I'm not really saying anything about continuity. – Varun Vejalla Aug 8 '19 at 14:48
• @JoonasIlmavirta Coming back to this answer after more than a year, could you expand on what you meant? I think I had misunderstood your comment before. – Varun Vejalla Aug 22 at 20:56
• What you computed is $\lim_{x\to0\pm}f'(x)$, whereas the one-sided derivatives at zero are $\lim_{h\to0\pm}h^{-1}[f(h)-f(0)]$. These are not the same thing! A function can be everywhere non-differentiable but it can still have one or both one-sided derivatives at zero. You observed that $f$ is differentiable whenever $x\neq0$ and that the limits at $x=0$ disagree. This only shows that if $f$ is differentiable everywhere, it cannot be continuous, unless you are implicitly appealing to a theorem that bridges the gap for you. – Joonas Ilmavirta Aug 23 at 8:46
• Ok, I think I understand now. I will update my answer accordingly. – Varun Vejalla Aug 23 at 17:07

Because it has a sharp corner at $$0$$, so it doesn't have a well defined tangent line; think about it, you can imagine many lines going through $$(0, 0)$$ that are tangent to the graph, so there are many possible tangent lines.

More formally, we have to investigate the limit

$$\lim_{h \to 0} \dfrac{f(0+h) - f(0)}{h}$$

This limit does not exist for the function, because if you let $$h$$ approach $$0$$ from the right, you get

$$\lim_{h \to 0^+} \dfrac {h-0}{h}=1.$$

While if you let $$h$$ approach $$0$$ from the left, that limit

$$\lim_{h \to 0^-} \dfrac {0-0}{h}=0.$$

Therefore the limit does not exist, so the function is not differentiable at $$0$$.

• "a tangent line is one which touches the graph only at one point" isn't entirely true. Consider the tangent of $x^2(x-1)$ at $x = 0$. It intersects the graph at $x = 1$. – Arthur Aug 8 '19 at 6:55
• I would not call that sharp corner or bend at $0$ a "cusp". A cusp is when the curve kind of turns 180 degrees so that the incoming tangent vector has the opposite direction of the outgoing tangent vector. The function $x\mapsto\sqrt{|x|}$ has a cusp. – Jeppe Stig Nielsen Aug 8 '19 at 7:27
• @Arthur Yeah I'm aware, but I think for the purposes of the OP it's fine. But I'm not actually sure of the definition of tangent line myself; the best thing I could think of is: "L is tangent to $f$ at $x$ if there exists some neighborhood around $x$ in which $L$ is either completely below or completely above the graph of $f$." This definition seems satisfactory to me from a geometric point of view; but then we would have to give up the notion that "derivatives give tangent lines," because for example $x^2 \sin (\frac 1x).$ – Ovi Aug 8 '19 at 13:44
• The tangent of $x^3$ at $x = 0$ then becomes problematic. I don't think you can get a proper definition of tangent without going into derivatives, or at least the same kinds of limits as the ones that define derivatives. – Arthur Aug 8 '19 at 13:47
• @JeppeStigNielsen Oh thanks for the info, I wasn't aware; my highschool teacher told us all of these sharp corners are cusps. I have fixed the answer. – Ovi Aug 8 '19 at 13:47