This question has to do with neural networks, but it is a purely mathematical question, so I think it belongs here.

Consider $f: \mathbb{R}^N \to \mathbb{R}^M$ such as to be implementable with a feedforward neural network with $H$ hidden layers and without biases, so:

$$ f(\vec{x}) = W_{H+2,H+1} \circ \sigma \circ W_{H+1,H} \circ \sigma \ldots \circ \sigma \circ W_{2,1} \vec{x}\tag{1}$$

where $\sigma$ is a non linearity (let's say a ReLU), and $W_{ij}$ are weights matricies (that connect layer $j$ with layer $i$). Does the hypothesis space change if we add neural bias on all the hidden neurons? In other words: if we modify (1) by adding vectors of parameters ($\vec{b_i}$) before each non linearity the space of functions that we can represent with $f$ (by changing the values of the parameters) changes in some way? I found no literature on the matter, some bibliographical reference would be much appreciated.


1 Answer 1


Yes, they are different.

Without biases, zero is always mapped to zero. More generally, you have $f(tx)=tf(x)$ for all $t\geq 0$. Of course this is specific to the ReLU.

With biases, this is not true anymore.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .