8
$\begingroup$

We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes. How many parameters does the network have?

$\endgroup$
3
  • $\begingroup$ What do you think makes up the parameters? $\endgroup$
    – Bram28
    Aug 26 '19 at 17:51
  • $\begingroup$ I mean the connection between any 2 units in a neural network can be interpreted as a parameter right? $\endgroup$
    – emily
    Aug 26 '19 at 17:52
  • $\begingroup$ That's exactly right. But there may be bias weights as well for each node. $\endgroup$
    – Bram28
    Aug 26 '19 at 19:18
12
$\begingroup$

Suppose the network has $784$ inputs, $16$ nodes in $2$ hidden layers and $10$ nodes in the output layer.

The amount of parameters (meaning weights and bias that make up the cost function) is then:

For the weights: $$784\times 16+16\times16+16\times10=12960$$

For the bias components:

We have $32$ neurons in the hidden layers and $10$ in the output, so we have $$32+10 = 42$$ biases.

So in total, the amount of parameters in this neural network is $13002$.

$\endgroup$
4
  • 1
    $\begingroup$ Thanks so much for the answer. That was exactly what I was looking for :) $\endgroup$
    – emily
    Aug 26 '19 at 18:08
  • $\begingroup$ @Steven31415 I think you would not add bias to output nodes. The total number of bias components should therefore be just 32, instead of 42. $\endgroup$
    – hrkrshnn
    Jan 12 '20 at 18:37
  • 3
    $\begingroup$ @jdoicj I disaggree on that because the output node also is a weighted sum with a bias. See also the video of 3blueonebrown called “But what is a neural network? Deep learning part 1” on 12min 21sec. $\endgroup$ Jan 12 '20 at 19:03
  • 1
    $\begingroup$ @Steven31415. My bad, I agree with you. Thanks for the reference. $\endgroup$
    – hrkrshnn
    Jan 12 '20 at 23:06
0
$\begingroup$

Let us say we have $l$ layers in a feed forward neural network, numbered from $0$ to $l-1$, with layer $0$ being input layer (no bias terms here as there are no neurons in this layer, they are simply input nodes). Let number of nodes in each layer be $n_0,n_1,n_2,n_3\dots n_{l-1}$. Then the number of parameters (weights) of the network including biases will be $$n_0\times n_1 + n_1\times n_2 + n_2\times n_3 + \dots + n_{l-2}\times n_{l-1} + ( n_1+n_2+n_3+\dots n_{l-1})\ .$$ The last expression (in bracket) is the number of bias terms in the network. This equals to the actual number of neurons in the feed forward network, as each neuron is having one bias input. Please note that giving bias input is not compulsory, although many libraries adds these inputs(typically fixed at 1) automatically to each neuron. It is important to note that input nodes are not neurons. Example for calculating number of weights(model parameters)of a Neural Network

The example above is a 2x2x2 network. according to the formula the number of model parameters(weights) of this Neural Network model = (2x2)+(2x2)+(2+2)=12.

$\endgroup$
-1
$\begingroup$

for simplicity lets say the $l=5$ clearly we have,

           i=h0        //inputs
           h=[h1,h2,h3,h4]   //hidden layers
           o=h5             //output

therefore the no of params $= (h0*h1 + h1*h2 + h2*h3 + h4*h5) + [h1+h2+h3+h4+h5]$ =click here to view the expression

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.