I have heard the following argument being made regarding Neural Networks:
- A Neural Network is a composition of several Activation Functions
- Sigmoid Activation Functions are Non-Convex Functions
- The composition of Non-Convex Functions can produce a Non-Convex Function
- Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex
Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):
e = 2.718
eq = function(x){ (-e^-x)* (1+e^-x)^-2 + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}
plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")
My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?
On it's own, the ReLU function is said to be Convex. Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function (The composition of two convex functions is convex). I understand that the composition of two Convex functions can produce a Concave Function - but still, the composition of two Convex Functions can not produce a "classic type of Non-Convex Function" (e.g. a function with several local minima and saddle points).
However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex. (The only thing I can think of is that the Loss Function in Neural Networks is made of linear combinations of function compositions - and even though compositions of convex functions are always convex, perhaps the linear combination of compositions for convex functions might not necessarily be convex ... but I am not sure about this)
Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?
Thanks!
References:
Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.
For example, below we can see the Loss Surfaces of (Modern) Neural Networks that clearly look Non-Convex: