I'm implementing a Perceptron network which basically consists of a single neuron in a single layer, trying to learn an OR logic port (linearly separable), but using the sigmoid function as activation:

$$ f(x) = \frac{1}{1 + e^{-\lambda x}} $$

and gradient descent to update the weights, adding to each weight:

$$ \Delta{w}_{i} = \eta \delta y(n), $$

where $n$ is the current data input (eg, $\{0,1\}$), $\eta$ is the learning rate, $y(n)$ is the output of the perceptron for input $n$, $t$ is the expected output for input $n$, and $\delta = f'(y(n))(t - y(n))$.

I'm using $f'(u) = u(1-u), u = f(y(n))$.

The activation part works pretty well, but the thing is, when I use the same activation function and use $\delta = t - y(n)$, instead of using gradient descent, it converges MUCH more quickly (around 300 iterations with precision of 0.0001 vs 100000 iterations with the same precision.

Increasing $\lambda$ makes it take more iterations to compute.

Could I be doing something wrong? I'm pretty sure gradient descent should converge much more quickly.

I'm using a bias input of weight 0.5 in both cases.

(I'm pretty new here, should I be posting this in Theoretical Computer Science instead?)



You must log in to answer this question.

Browse other questions tagged .