1
$\begingroup$

In machine learning (particularly with regards to Neural Nets), there's a bunch of "ramp" functions that're used. For example, the ReLU is $0$ for $x\leq 0$ and $x$ for $x>0$.

I was trying to find a nice smooth ramp that was $\sim x$ as $x\to\infty$ and $\sim\frac{1}{-x}$ as $x\to-\infty$. And I came up with

$$ \text{SmoothRamp}(x)=\begin{cases} \ x+1 & x\geq 0,\\ \frac{1}{1-x} & x<0. \end{cases} $$

This ramp also has the nice properties of being differentiable, having $f(-x) = \frac{1}{f(x)}$, and being easily computable.

Anyways, my question is: is this smooth ramp already a thing with a common name? Or is it something that I made up?

$\endgroup$
1
  • 1
    $\begingroup$ In the image that I found in here, towardsdatascience.com/…, it doesn't seem to be in use, and in a springer article "Quantitative Estimates for Neural Network Operators Implied by the Asymptotic Behaviour of the Sigmoidal Activation Functions," the activation functions considered are $O(|x|^{-\alpha})$ for $\alpha > 1$ as $x \to -\infty$. I'm not an expert but it would appear what you made is new, but I'm not sure if people do/can do analysis on the convergence of a NN with this operator? $\endgroup$
    – 1mdlrjcmed
    Sep 6 at 21:30

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .