# Neural Networks: Different Depths and Widths But Same Number of Parameters

Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of parameters but with different depths and widths. The solution to this problem gets reduced to a system of equations with integer solutions. Nonetheless, I haven't come up with an idea to solve it, even by using numerical methods. Therefore, I wanted to ask if there's someone that could help with finding a solution.

Context

A sequential Neural Network is defined by a number of $$n$$ matrices that correspond to the weights of each layer ($$\{W_i\}_{i = 1,\dots, n}$$). The shape of each consecutive matrix is directly related to the previous one (since they are composed after applying a non-linear function) such that: $$W_i \in \mathbb{R}^{a\times b} \Leftrightarrow W_{i+1} \in \mathbb{R}^{b\times c}$$ where $$a, b, c \in \mathbb{N}$$.

On the other hand, the number of parameters of a Neural Network is the same as the sum of the total number of entries of each matrix $$W_i$$. Assuming that the number of neurons in the input layer and the number of neurons in the output layer remain constant (let's call them $$a$$ and $$c\in \mathbb{N}$$) and the number of neurons the same for each hidden layer (let's call it $$b_i \in \mathbb{N}$$ for a network with $$i$$ hidden layers) we have that we can calculate the number of parameters in the network. For example, for a network with one hidden layer, the number of parameters would be such that: $$p_1 = ab_1 + b_1c$$ Then, if we add another hidden layer, the number of parameters would be such that: $$p_2 = ab_2 + b_2b_2 + b_2c \\ = ab_2 + b_2^2 + b_2c$$ Therefore, for the general case with $$m$$ hidden layers the number of parameters is defined by the equation: $$p_m = ab_m + \sum_{i=1}^{m}{b_m^2} + b_mc \\ = ab_m + mb_m^2 + b_mc \\ = \boxed{b_m (a + mb_m + c)}$$

Question

In my case, since I want to create $$m$$ neural networks ($$5 \leq m \leq 12$$ would be enough for the purposes of my research) I would need to find a solution to find the $$b_i$$'s $$\in \mathbb{N}$$ such that all of the $$m$$ networks have the same number of parameters $$P$$, where $$P$$ can be any positive integer (hopefully the smallest one where the system has solutions). $$\begin{cases} P = ab_1 + b_1^2 + cb_1\\ P = ab_2 + 2b_2^2 + cb_2 \\ \dots \\ P = ab_m + mb_m^2 + cb_m \\ \end{cases}$$ $$\Leftrightarrow$$ $$\begin{cases} P = b_1 (a + b_1 + c)\\ P = b_2 (a + 2b_2 + c) \\ \dots \\ P = b_m (a + mb_m + c) \\ \end{cases}$$ Is there a closed-form solution for this system of equations in the positive integers? That is, a solution such that we could express the set of $$b_i$$'s in terms of $$a, c,$$ and $$m$$ where $$b_i \in \mathbb{N}, \forall i \in \{1, \dots, m\}$$?

I would really appreciate any help. Thank you all in advance.

Edit: As @LutzLehmann pointed out, relaxing the constraints for the number coefficients to be in a band of 1% or 5% of $$P$$ would serve too as a solution for the problem. Regarding relaxing the number of neurons in the hidden layers to allow them to be different for the different layers, I think it would be useful for the project if the difference between the number of neurons in each layer do not differ in a great extent.

• If you relax the constraints, allowing each layer to have a potentially different number of neurons with an overall shape like $(a, b_1, b_2,b_3,\ldots, b_m, c)$, $P$ is calculated by the dot product of the first $m+1$ terms with the last $m+1$ terms. Exhaustive search shows the number of distinct shapes $S$ for $P = 1,2,3,\ldots$ is $S = 1,3,5,10,14,27,\ldots$ which does not appear to be in the Online Encyclopedia of Integer Sequences oeis.org/search?q=1%2C3%2C5%2C10%2C14%2C27 example: $P=5$ gives $[[1,4],[2,2],[4,1],[1,1,3],[1,2,1],[2,1,2],[3,1,1],[1,1,1,2],[2,1,1,1],[1,1,1,1,1]]$ Jun 23, 2022 at 17:25
• I would keep the number of coefficients inside a band of 1% or 5% width to avoid the requirement of exact integer solutions. Jun 23, 2022 at 17:32
• import Control.Monad (replicateM) shapes p = [ s | m <- [0..p], s <- replicateM (m + 2) [1..p], p == sum (zipWith (*) s (tail s)) ] main = mapM_ print $shapes (8 :: Int) is my highly inefficient$O(p^p)$search program in Haskell Jun 23, 2022 at 17:58 • Thank you for the recommendations. Since for the purposes of my research$P$would be relatively large ($P > 100$) it would be somewhat infeasible to use an exhaustive search for the solution. However, the proposal of relaxing the constraints is reasonable, only if the$b_i$'s do not get too close to$1$and have similar values since otherwise that would create a bottleneck in the network affecting the results. @LutzLehmann, what method would you recommend to solve the system inside a band as you proposed? I think that idea would work. Jun 23, 2022 at 18:16 • Solve the quadratic equation rounding up, then reduce successively the node counts by 1 starting from the output side until the best fit is reached. Jun 25, 2022 at 9:16 ## 1 Answer Assume all hidden layers have exactly the same size $$b_m$$. By the quadratic formula applied to your equation with a box, considering $$b_m$$ should be positive, one gets: $$b_m \approx \hat{b_m} = \frac{\sqrt{(a + c)^2 + 4 m P} - (a + c)}{2 m}$$ where exact equality is unlikely. Pick $$P$$ and $$m$$, calculate and pick a nearby integer for $$b_m$$ as above, then calculate $$P' = b_m (a + m b_m + c).$$ For example, classifying the famous Iris data set with $$a = 4, c = 3, P = 1000$$ I calculated the following table: m b_m P' %oo 5 15 1005 50 6 13 936 -640 7 12 948 -520 8 11 924 -760 9 11 1045 450 10 10 970 -300 11 10 1070 700 12 9 954 -460 where the last column shows they are all within a ±8% interval, represented as ±800 in the table. With $$a = 28^2 = 784, c = 10, P = 100000$$ (for example, a MNIST digit image classifier) I get: m b_m P' %oo 5 83 100347 35 6 79 100172 17 7 76 100776 78 8 73 100594 59 9 70 99680 -32 10 68 100232 23 11 66 100320 32 12 64 99968 -3 satisying ±1% tolerance (which would be ±100 in the table). I used this Haskell code: p = ... a = ... c = ... b m = fromInteger . round$
(sqrt((a + c)^2 + 4 * m * p) - (a + c)) / (2 * m)
p' m = b m * (a + m * b m + c)

main = mapM_ putStrLn $$[ intercalate "|" . ([""]++) . (++[""]) . map (show . round)$$ [ m, b m, p' m, 10000 * (p' m - p) / p ]
| m <- [5..12]
]


Possibly better tolerances could be achieved for lower P values by using both nearby integers near $$\hat{b_m}$$ spread between different layers.