We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes. How many parameters does the network have?
Suppose the network has 784 inputs, 16 nodes in two hidden layers and 10 nodes in the output layer. The amount of parameters (meaning weights and bias that make up the cost function) is then: 784*16+16*16+16*10 for the weights, which is 12960. We have 32 neurons in the hidden layers and 10 in the output, so in total 32+10 = 42 bias components. So in total, the amount of parameters in this neural network is 13002.