Good Day

I am trying to get an understanding of Neural Network. Have gone through few web sites. Came to know the following:

1) One of main objective of neural network is to “predict” based on data. 2) To predict a. Train the network with known data b. Calculate weights by finding difference between “Target Output” and “Calculated Output”. c. To do that we use derivative, partial derivative(chain rule etc..)

I can understand the overall concept of neural network a) I can also understand “Derivative” is nothing but Rate of change of one quantity over another(at a given point). b) Partial derivative is Rate of change of one quantity over another, irrespective of another quantity , if more than two factors are in equation.

The point that I canNOT relate or understand clearly is, a) why should we use derivative in neural network, how exactly does it help b) Why should we activation function, in most cases its Sigmoid function. c) I could not get a complete picture of how derivatives helps neural network.

Can you guys please help me understand the complete picture, iff possible try not to use mathematical terms, so that it will be easy for me to grasp.

Thanks, Satheesh

  • $\begingroup$ Forget the neural networks and look instead at the gradient descent. Once you got it (and applied it to simple problems) you have to understand how the partial derivatives are computed in a neural network (the backpropagation) and simply use those for the gradient descent. $\endgroup$
    – reuns
    Commented Jun 22, 2016 at 16:47
  • $\begingroup$ Thank you, I am trying to understand gradient descent. $\endgroup$ Commented Jun 22, 2016 at 23:45
  • $\begingroup$ There is a cost function $L$ that we're minimizing which quantifies how accurate our predictions are on the training dataset. We want $L$ to be small (because a smaller value of $L$ corresponds to more accurate predictions). So we minimize $L$ using gradient descent. The vector $-\nabla L(\beta)$ points in the direction of steepest descent for $L$, and the idea behind gradient descent is to repeatedly take short steps in the direction of steepest descent, reducing the value of $L$ a little bit at each step. $\endgroup$
    – littleO
    Commented May 26, 2020 at 17:13

2 Answers 2


As you said: "Partial derivative is Rate of change of one quantity over another, irrespective of another quantity , if more than two factors are in equation."

It means that we can measure the rate of change of the output error w.r.t. network weights. If we know how the error changes w.r.t. weights, we can change those weights in a direction that decreases the error. But as @user1952009 said, it is just gradient descent. Neural networks combine it with the chain rule to update non-output layers.

Regarding sigmoid activations, it has 2 uses: 1) to bound the neuron output; 2) to introduce nonlinearities into the network. This last item is essential to make the neural network solve problems not solvable by simple linear/logistic regression. If neurons hadn't nonlinear activation functions, you could rewrite your entire network as a single layer, which is not as useful. For instance, suppose a 2-layer neural network. Its output would be $y = W_o(W_i\mathbf{x})$ ($W_i$ = input weights, $W_o$ = output weights, $\mathbf{x}$ = input), which can be rewritten as $y = (W_oW_i)\mathbf{x}$. Let $W = W_oW_i$, it leaves us with a single layer neural network $y = W\mathbf{x}$.

  • $\begingroup$ Hello Professor - Thanks much for taking time to answer. By any chance can you please point me to some examples ? with detailed explanation. $\endgroup$ Commented Jun 22, 2016 at 18:44
  • $\begingroup$ Thanks, I did not complete my last question, added by mistake. I will post my question shortly. $\endgroup$ Commented Jun 22, 2016 at 18:59
  • $\begingroup$ @Satheesh : so you don't know matrices/vectors/linear algebra neither ? And in rcpinto answer it is important the fact that in $y_j = f(\sum_i W_{i,j} y_i)$ the output of the $j$th neuron in term of the output of the other neurons, the activation function is chosen to be bounded because otherwise it would be easy to have a non-stable network. And the sigmoid function is preferred because it acts as a boolean function (so that the network can simulate any (cycle-less) logic circuit ) $\endgroup$
    – reuns
    Commented Jun 23, 2016 at 0:00
  • $\begingroup$ @rcpinto Thanks for your reply. Yes, not an expert. If you guys can please help with your expertise to solve this, I might be able to understand better, Just high level is fine. Just an example, so that I can understand. Let’s just say I want a basic NN to predict value of a house per square foot. I have the following data for past 10 yrs. 1) No of schools in that zip 2) School Rating in that zip 3) Zip 4) No of bedroom 5) Size of backyard 6) Price per square foot... Continued in next comment $\endgroup$ Commented Jun 24, 2016 at 0:18
  • Partial Derivative comes into play because we train neural network with gradient descent, which involves partial derivative when dealing with multivariable case
  • In the final output layer, you can do a sigmoid transformation or tanh or ReLu or nothing at all! It all depends on you. That flexibility is exactly what makes neural networks so powerful in expression capability.

In fact, neural works are nothing but a fancy, popular nonlinear estimator.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .