# How many parameters does the neural network have?

We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes. How many parameters does the network have?

• What do you think makes up the parameters? Aug 26 '19 at 17:51
• I mean the connection between any 2 units in a neural network can be interpreted as a parameter right? Aug 26 '19 at 17:52
• That's exactly right. But there may be bias weights as well for each node. Aug 26 '19 at 19:18

Suppose the network has $$784$$ inputs, $$16$$ nodes in $$2$$ hidden layers and $$10$$ nodes in the output layer.

The amount of parameters (meaning weights and bias that make up the cost function) is then:

For the weights: $$784\times 16+16\times16+16\times10=12960$$

For the bias components:

We have $$32$$ neurons in the hidden layers and $$10$$ in the output, so we have $$32+10 = 42$$ biases.

So in total, the amount of parameters in this neural network is $$13002$$.

• Thanks so much for the answer. That was exactly what I was looking for :) Aug 26 '19 at 18:08
• @Steven31415 I think you would not add bias to output nodes. The total number of bias components should therefore be just 32, instead of 42. Jan 12 '20 at 18:37
• @jdoicj I disaggree on that because the output node also is a weighted sum with a bias. See also the video of 3blueonebrown called “But what is a neural network? Deep learning part 1” on 12min 21sec. Jan 12 '20 at 19:03
• @Steven31415. My bad, I agree with you. Thanks for the reference. Jan 12 '20 at 23:06

Let us say we have $$l$$ layers in a feed forward neural network, numbered from $$0$$ to $$l-1$$, with layer $$0$$ being input layer (no bias terms here as there are no neurons in this layer, they are simply input nodes). Let number of nodes in each layer be $$n_0,n_1,n_2,n_3\dots n_{l-1}$$. Then the number of parameters (weights) of the network including biases will be $$n_0\times n_1 + n_1\times n_2 + n_2\times n_3 + \dots + n_{l-2}\times n_{l-1} + ( n_1+n_2+n_3+\dots n_{l-1})\ .$$ The last expression (in bracket) is the number of bias terms in the network. This equals to the actual number of neurons in the feed forward network, as each neuron is having one bias input. Please note that giving bias input is not compulsory, although many libraries adds these inputs(typically fixed at 1) automatically to each neuron. It is important to note that input nodes are not neurons.

The example above is a 2x2x2 network. according to the formula the number of model parameters(weights) of this Neural Network model = (2x2)+(2x2)+(2+2)=12.

for simplicity lets say the $$l=5$$ clearly we have,

           i=h0        //inputs
h=[h1,h2,h3,h4]   //hidden layers
o=h5             //output


therefore the no of params $$= (h0*h1 + h1*h2 + h2*h3 + h4*h5) + [h1+h2+h3+h4+h5]$$ =click here to view the expression