Neural Network Sigmoid Problem

Question

I currently am trying to create a three layer neural network and I want to use the back propagation.

My training data is:

Input : Output
0 | 1 : 0
1 | 1 : 2

In many articles that I've read, they use

          1
f(x) = ---------
       1 + e-input

to calculate the output of each neuron.

But this function can only return an output between 0.5 and 1, right? Does anyone know another function that I can use for calculating the output?

Answer 1

The important thing to understand here is not the range of the sigmoid function, but instead what kind of functions can be used here. The basic idea is that you want a function which (even if after normalization), can act like a "yes-no decision" or as Jair Taylor said in his/her answer above whether the neuron "fires" or not.

These functions are called "activation functions" because they can be interpreted as how much this particular neuron of the layer was activated by the input function.

Some common functions used for this purpose are sigmoid(x), Tanh(x) and the rectified linear function (used more in deep learning literature).

To get hold of some theory on this, check out the CS231n lectures on github hosted by Stanford University. Hope it helps!

Answer 2

The sigmoid returns a value between $0$ and $1$. This is sometimes interpreted as a probability, or as a measure of whether the neuron "fires" or not. If you don't want to output a probability in the end, you could just not put a sigmoid on the last layer.