Paradox of square error function and derivates in neural networks

Square error function in neural network is defined as square of (target-output). This number is always positive, because of square.

On the other side, derivatives of most common functions like Sigmoid or Relu are 0 or more, they are not negative.

My question is. How can we update weights in negative direction (if needed) using Sigmoid or Relu with square error function, if you can not change the value of weights lower to zero or less than zero, assuming these are randomly set up above zero.

1 Answer

Consider a toy loss function: $$L(w)=(\sigma(wx)-y)^2$$

Its derivative is: $$L'(w)=2(\sigma(wx)-y)\cdot\sigma'(wx)\cdot x$$

Although $\sigma'$ is nonnegative, the other terms $(\sigma(wx)-y)$ and/or $x$ could have any sign. So if you update in the direction of $-L'(w)$, you might go in any direction.

• I am close to enlightenment. So in neural network there are two important derivatives. First, the derivative of activation function and second derivative of loss function. Within derivative of loss function is derivative of activation function. Is this correct? – Testing man Mar 8 '18 at 20:20
• That's a good way to put it! We ultimately care about the derivative of the loss function. Because of the chain rule, that involves other derivatives, including the activation function. And the precise statement of the chain rule tells you how to set up back-propagation. – Chris Culter Mar 8 '18 at 20:40
• Thank you very much Chris! – Testing man Mar 8 '18 at 20:49