We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes. How many parameters does the network have?

  • $\begingroup$ What do you think makes up the parameters? $\endgroup$ – Bram28 Aug 26 '19 at 17:51
  • $\begingroup$ I mean the connection between any 2 units in a neural network can be interpreted as a parameter right? $\endgroup$ – emily Aug 26 '19 at 17:52
  • $\begingroup$ That's exactly right. But there may be bias weights as well for each node. $\endgroup$ – Bram28 Aug 26 '19 at 19:18

Suppose the network has 784 inputs, 16 nodes in two hidden layers and 10 nodes in the output layer. The amount of parameters (meaning weights and bias that make up the cost function) is then: 784*16+16*16+16*10 for the weights, which is 12960. We have 32 neurons in the hidden layers and 10 in the output, so in total 32+10 = 42 bias components. So in total, the amount of parameters in this neural network is 13002.

  • $\begingroup$ Thanks so much for the answer. That was exactly what I was looking for :) $\endgroup$ – emily Aug 26 '19 at 18:08
  • $\begingroup$ No problem. Glad it helped. Don't forget to accept an answer when done. Grts $\endgroup$ – Steven31415 Aug 26 '19 at 18:09
  • $\begingroup$ @emily can you accept the answer so this can be flagged as Solved ? $\endgroup$ – Steven31415 Dec 24 '19 at 13:31
  • 1
    $\begingroup$ @jdoicj I disaggree on that because the output node also is a weighted sum with a bias. See also the video of 3blueonebrown called “But what is a neural network? Deep learning part 1” on 12min 21sec. $\endgroup$ – Steven31415 Jan 12 at 19:03
  • 1
    $\begingroup$ @Steven31415. My bad, I agree with you. Thanks for the reference. $\endgroup$ – jdoicj Jan 12 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.