1
$\begingroup$

I was looking at the following function (called "ReLU") :

enter image description here

I am trying to understand why this function ("ReLU") is considered to be non-linear, when it appears to look "piecewise linear" (and even contains the term "linear" in its name):

  • Can someone please explain why the "ReLU" function is described as non-linear, when it seems to be linear in appearance? Is it possible that the individual "pieces" of the ReLU function are linear, but the entire function itself is somehow non-linear?

  • When functions are defined in "pieces" - can we still determine if the entire function is "convex" or "non-convex" - or are we forced to only label the individual pieces of the function as convex and non-convex?

Thanks!

References:

$\endgroup$
7
  • 3
    $\begingroup$ To see why the function is said to not be linear, consider the definition of "linear." In the context of linear algebra, linear means $R(x+y)=R(x)+R(y)$, which is not true for this function $R$. In the context of elementary algebra, it means the graph is a line, which is also not true (it is multiple line segments / rays, not a line). Bottom line: words actually mean things, and "linear" is a word. $\endgroup$
    – runway44
    Jan 24 at 22:42
  • 2
    $\begingroup$ If a piece of a function's graph (i.e. the function's restriction to a smaller interval) is not convex, then neither is the function as a whole. The converse is not true: even if all "pieces" of a function are convex, that does not guarantee the function as a whole is convex. That means more work needs to be done to determine if such a function as a whole is convex or not. $\endgroup$
    – runway44
    Jan 24 at 22:47
  • $\begingroup$ "When functions are defined in "pieces"" $\;-\;$ It is not useful to think of function definitions as being "in pieces" vs. not. A function is the same function no matter how you write its definition. You can define for example $|x|=\sqrt{x^2}$ in one piece, or you can write the definition piecewise for $\mathbb R^-, \mathbb R^+$ but it's still the same function in either case. And, like any local behaviors, a property that applies to only part of the domain does not necessarily apply to the entire domain e.g. $\sqrt{|x|}$ is concave on $\mathbb R^-,\mathbb R^+$ but not on $\mathbb R$. $\endgroup$
    – dxiv
    Jan 25 at 0:15
  • $\begingroup$ @runway44: Surely if all "pieces" of a function are convex, then the function is convex? $\endgroup$
    – TonyK
    Jan 25 at 0:19
  • $\begingroup$ @TonyK Not necessarily, consider for example $\,f(x) = \begin{cases}\begin{align}e^{-x} & & x \ge 0 \\ e^{x} & & x \lt 0\end{align}\end{cases}\,$. $\endgroup$
    – dxiv
    Jan 25 at 0:27

1 Answer 1

4
$\begingroup$

There are two competing definitions for a linear function $f:\Bbb R\mapsto\Bbb R$, but neither of them would allow ReLU.

The first definition most are exposed to is that a linear function is polynomial of degree 1, i.e. a function in the form

$$ f(x) = ax + b $$

(as Brian Borchers points out in the comments, this type of function is usually called affine beyond elementary algebra to avoid conflict with the second definition below)

The second definition comes from linear algebra, and is a restriction on this class of functions. We say a function is linear iff

$$ f(x+\alpha y) = f(x) + \alpha f(y) $$

for $x,y$ from some vector space and $\alpha$ from the underlying field. In this case, the vector space and the underlying field are both $\Bbb R$, and this implies the form

$$ f(x) = ax $$

for all such functions.

Any attempt to put ReLU into this form will result in a function which is only valid on a half-line. To agree with the left half-line, we need $f(x) \equiv 0$; for the right half-line, we need $f(x) = x$.

Because there is a partition of the line into intervals such that ReLU is linear (in both senses) on the interior of each interval, we say that ReLU is piecewise linear. Obviously, any linear function is piecewise linear, but the same is not true in reverse, as we see here with ReLU.

So, to directly answer your question, "piecewise linear" should not be interpreted as "nonlinear" in general, but there are piecewise linear functions which are nonlinear (and those which are linear).


As with linearity, we can discuss convexity in both global and piecewise contexts.

A function is called convex on some domain when every pair of points $x,y$ in that domain has

$$ f(\theta x + (1-\theta)y) \leq \theta f(x) + (1-\theta)f(y) $$

for all $\theta\in [0,1]$.

Linear functions (in both senses) are convex, and so piecewise linear implies piecewise convex. Thus, ReLU is (at least) piecewise convex. However, we can directly verify that ReLU is globally convex as well by substituting the definition into the above inequality.

$\endgroup$
4
  • 2
    $\begingroup$ Functions of the form $f(x)=ax+b$ are called "affine" to distinguish them from the other kind of linear function. $\endgroup$ Jan 25 at 0:09
  • 1
    $\begingroup$ @BrianBorchers I agree in principal, but there are enough references that use "linear" to refer to affine functions on $\Bbb R$ (likely because their graphs are lines) that it warrants consideration. The purpose of the inclusion in this answer was to show that even the more permissive idea of linear functions fails to include ReLU. $\endgroup$
    – Alex Jones
    Jan 25 at 0:23
  • $\begingroup$ @ Alex Jones: Thank you so much for your answer! Are we also able to determine if the ReLU function is "convex"? thank you so much! $\endgroup$
    – stats_noob
    Jan 25 at 3:38
  • $\begingroup$ @stats555 I added a brief discussion of convexity to the answer, but in short: ReLU is indeed globally convex. $\endgroup$
    – Alex Jones
    Jan 25 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.