2
$\begingroup$

This question has to do with neural networks, but it is a purely mathematical question, so I think it belongs here.

Consider $f: \mathbb{R}^N \to \mathbb{R}^M$ such as to be implementable with a feedforward neural network with $H$ hidden layers and without biases, so:

$$ f(\vec{x}) = W_{H+2,H+1} \circ \sigma \circ W_{H+1,H} \circ \sigma \ldots \circ \sigma \circ W_{2,1} \vec{x}\tag{1}$$

where $\sigma$ is a non linearity (let's say a ReLU), and $W_{ij}$ are weights matricies (that connect layer $j$ with layer $i$). Does the hypothesis space change if we add neural bias on all the hidden neurons? In other words: if we modify (1) by adding vectors of parameters ($\vec{b_i}$) before each non linearity the space of functions that we can represent with $f$ (by changing the values of the parameters) changes in some way? I found no literature on the matter, some bibliographical reference would be much appreciated.

$\endgroup$

1 Answer 1

4
$\begingroup$

Yes, they are different.

Without biases, zero is always mapped to zero. More generally, you have $f(tx)=tf(x)$ for all $t\geq 0$. Of course this is specific to the ReLU.

With biases, this is not true anymore.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .