# Is it necessary to use sigmoid function in every layer of neural network?

Suppose I have a neural network that has one input layer, one hidden layer, and a single output.
As far as I know, the sigmoid function is used for keeping the output between 0-1 (i.e. 0-100%). But is it necessary to use it in every layer? Suppose I have calculated the value of the second layer without using the sigmoid function. But have used it for calculating the third layer (which is output). Will there be any problem?

• No! It would be very unusual and probably bad to do that. But you should use an activation function such as ReLU in every layer of your network. (There needs to be some nonlinearity in each layer, because the composition of two linear functions is just another linear function.) Oct 15, 2020 at 16:33
• Further to @littleO's point, the activation function acts after a linear one viz. $a(Wx+b)$, so if the last activation function is linear it may as well be the identity and that layer may as well not exist, and if some activation function other than the last one is linear the first three of the four steps "linearly transform, apply a linear activation, linearly transform, apply a nonlinear activation" collapse to just "linearly transform", so again the linear layer is effectively nonexistent.
– J.G.
Oct 15, 2020 at 16:54
• I guess in my comment above I forgot the history of neural networks -- before ReLU, it was common to use the sigmoid activation function in every layer. But then it was discovered that ReLU tends to work much better (perhaps because it helps avoid the "vanishing gradient" problem). Oct 15, 2020 at 17:36

Mathematically, artificial neural networks are just mathematical functions. You can apply whatever function you want for each neuron it is still a function. If you want to apply the sigmoid function only in the last layer you can do that. However, activation functions have a certain purpose. They make a neural network more powerful. Observe that a composition of affine functions (i.e., functions of the form $$f(x)= ax +b$$) is still affine, hence, if you don't apply an activation function in your hidden layers at all, you basically render them useless. It is the same as if there would be no hidden layers at all. That is why we use non-linear activation functions. Esentially, every neuron that doesn't have a non-linear activation function is useless.
In short, it is not necessary to use the sigmoid function at every layer. Check out Wikipedia for activation functions, sigmoid function and identity function are both activation functions. The main difference is that sigmoid function has range in $$[0,1]$$ and is nonlinear, whereas identity function is linear and has no restriction for its output. From my understanding, not using the sigmoid function ($$i.e.$$ using the identity function) in the second layer of your output should work fine.