I've just read the following

The basic unit ("neuron" i) performs the following computation to update its state $y_i$: it computes a weighted sum $v_i$ of its inputs $x:j$ which is passed through a sigmoid squashing function $g ( \cdot )$.

Source: Design of a neural network character recognizer for a touch terminal

I know what a sigmoid function is, but what is a sigmoid squashing function? I have also seen this in the PyBrain documentation.

  • $\begingroup$ It's basically a sigmoid function used to compress the outputs. $\endgroup$ – Silynn Jun 18 '14 at 21:32
  • $\begingroup$ So is "squashing function" a synonym for "activation function"? $\endgroup$ – Martin Thoma Jun 18 '14 at 21:35
  • $\begingroup$ So is it "a squashing function that we chose to be a sigmoid function" or is it "a special type of sigmoid function that is called 'squashing'"? $\endgroup$ – Martin Thoma Jun 18 '14 at 21:36
  • $\begingroup$ They just mean that they squash the signal (saturate it), using a sigmoid function (as you could use other functions for squashing). $\endgroup$ – Yves Daoust Jun 18 '14 at 22:11

The sigmoid squashing function is the same as the sigmoid function.

The term sigmoid squashing function is favored in the neural net community. The logistic function is the classical squashing function. Other sigmoid functions include: arctangent, the hyperbolic tangent, the Gudermannian function, and the error function.

It is called the squashing function because it is tightly bounded to the pair of horizontal asymptotes. Thus, useful in compressing, or squashing, outputs.

| cite | improve this answer | |
  • $\begingroup$ Do you mean something like this? Given an input, its effect is (let's say) linear when the activation to which it will contribute is (e.g.) near 0 (for activations in [-1,1]), but when the activation is near an extremum (1, let's say), then the effect of the input is reduced--i.e. it is "squashed". $\endgroup$ – Mars Jan 26 '15 at 19:14

Fix a reference measure $\mu$.
A sigmoid function $\sigma_0$ is a essentially-bounded map from $\mathbb{R}$ to itself. I.e.: $\sigma_0 \in L^{\infty}_{\mu}(\mathbb{R})$.

While a squashing function $\sigma$ is a sigmoid function satisfying $$ \operatorname{ess-sup}|\sigma| \leq 1 , $$ thus, $\sigma \in Ball_{L^{\infty}_{\mu}(\mathbb{R})}$.

Practical Lingo:

In practical terms, if $\mu$ is the Lebesgue measure, then $\sigma_0:\mathbb{R}\to [m,M]$ for some $m\leq M$, real numbers, while $\sigma:\mathbb{R} \to [-1,1]$.

I think this paper has a rigerous definitio (under the additional assumption of continuity). While this paper hints to the non-continuous version.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.