14
$\begingroup$

We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes. How many parameters does the network have?

$\endgroup$
3
  • $\begingroup$ What do you think makes up the parameters? $\endgroup$
    – Bram28
    Commented Aug 26, 2019 at 17:51
  • $\begingroup$ I mean the connection between any 2 units in a neural network can be interpreted as a parameter right? $\endgroup$
    – emily
    Commented Aug 26, 2019 at 17:52
  • $\begingroup$ That's exactly right. But there may be bias weights as well for each node. $\endgroup$
    – Bram28
    Commented Aug 26, 2019 at 19:18

3 Answers 3

29
$\begingroup$

Suppose the network has $784$ inputs, $16$ nodes in $2$ hidden layers and $10$ nodes in the output layer.

The amount of parameters (meaning weights and bias that make up the cost function) is then:

For the weights: $$784\times 16+16\times16+16\times10=12960$$

For the bias components:

We have $32$ neurons in the hidden layers and $10$ in the output, so we have $$32+10 = 42$$ biases.

So in total, the amount of parameters in this neural network is $13002$.

$\endgroup$
5
  • 1
    $\begingroup$ Thanks so much for the answer. That was exactly what I was looking for :) $\endgroup$
    – emily
    Commented Aug 26, 2019 at 18:08
  • $\begingroup$ @Steven31415 I think you would not add bias to output nodes. The total number of bias components should therefore be just 32, instead of 42. $\endgroup$
    – hrkrshnn
    Commented Jan 12, 2020 at 18:37
  • 3
    $\begingroup$ @jdoicj I disaggree on that because the output node also is a weighted sum with a bias. See also the video of 3blueonebrown called “But what is a neural network? Deep learning part 1” on 12min 21sec. $\endgroup$ Commented Jan 12, 2020 at 19:03
  • 1
    $\begingroup$ @Steven31415. My bad, I agree with you. Thanks for the reference. $\endgroup$
    – hrkrshnn
    Commented Jan 12, 2020 at 23:06
  • 1
    $\begingroup$ @emily you should accept this answer, then. $\endgroup$
    – Umagon
    Commented Feb 18, 2022 at 5:06
1
$\begingroup$

Let us say we have $l$ layers in a feed forward neural network, numbered from $0$ to $l-1$, with layer $0$ being input layer (no bias terms here as there are no neurons in this layer, they are simply input nodes). Let number of nodes in each layer be $n_0,n_1,n_2,n_3\dots n_{l-1}$. Then the number of parameters (weights) of the network including biases will be $$n_0\times n_1 + n_1\times n_2 + n_2\times n_3 + \dots + n_{l-2}\times n_{l-1} + ( n_1+n_2+n_3+\dots n_{l-1})\ .$$ The last expression (in bracket) is the number of bias terms in the network. This equals to the actual number of neurons in the feed forward network, as each neuron is having one bias input. Please note that giving bias input is not compulsory, although many libraries adds these inputs(typically fixed at 1) automatically to each neuron. It is important to note that input nodes are not neurons. Example for calculating number of weights(model parameters)of a Neural Network

The example above is a 2x2x2 network. according to the formula the number of model parameters(weights) of this Neural Network model = (2x2)+(2x2)+(2+2)=12.

$\endgroup$
0
$\begingroup$

for simplicity lets say the $l=5$ clearly we have,

           i=h0        //inputs
           h=[h1,h2,h3,h4]   //hidden layers
           o=h5             //output

therefore the no of params $= (h0*h1 + h1*h2 + h2*h3 + h4*h5) + [h1+h2+h3+h4+h5]$ =click here to view the expression

$\endgroup$
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .