Square error function in neural network is defined as square of (target-output). This number is always positive, because of square.

On the other side, derivatives of most common functions like Sigmoid or Relu are 0 or more, they are not negative.

My question is. How can we update weights in negative direction (if needed) using Sigmoid or Relu with square error function, if you can not change the value of weights lower to zero or less than zero, assuming these are randomly set up above zero.


Consider a toy loss function: $$L(w)=(\sigma(wx)-y)^2$$

Its derivative is: $$L'(w)=2(\sigma(wx)-y)\cdot\sigma'(wx)\cdot x$$

Although $\sigma'$ is nonnegative, the other terms $(\sigma(wx)-y)$ and/or $x$ could have any sign. So if you update in the direction of $-L'(w)$, you might go in any direction.

  • $\begingroup$ I am close to enlightenment. So in neural network there are two important derivatives. First, the derivative of activation function and second derivative of loss function. Within derivative of loss function is derivative of activation function. Is this correct? $\endgroup$ Mar 8 '18 at 20:20
  • 1
    $\begingroup$ That's a good way to put it! We ultimately care about the derivative of the loss function. Because of the chain rule, that involves other derivatives, including the activation function. And the precise statement of the chain rule tells you how to set up back-propagation. $\endgroup$ Mar 8 '18 at 20:40
  • $\begingroup$ Thank you very much Chris! $\endgroup$ Mar 8 '18 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.