# Is this smooth linear ramp function already a thing?

In machine learning (particularly with regards to Neural Nets), there's a bunch of "ramp" functions that're used. For example, the ReLU is $$0$$ for $$x\leq 0$$ and $$x$$ for $$x>0$$.

I was trying to find a nice smooth ramp that was $$\sim x$$ as $$x\to\infty$$ and $$\sim\frac{1}{-x}$$ as $$x\to-\infty$$. And I came up with

$$\text{SmoothRamp}(x)=\begin{cases} \ x+1 & x\geq 0,\\ \frac{1}{1-x} & x<0. \end{cases}$$

This ramp also has the nice properties of being differentiable, having $$f(-x) = \frac{1}{f(x)}$$, and being easily computable.

Anyways, my question is: is this smooth ramp already a thing with a common name? Or is it something that I made up?

• In the image that I found in here, towardsdatascience.com/…, it doesn't seem to be in use, and in a springer article "Quantitative Estimates for Neural Network Operators Implied by the Asymptotic Behaviour of the Sigmoidal Activation Functions," the activation functions considered are $O(|x|^{-\alpha})$ for $\alpha > 1$ as $x \to -\infty$. I'm not an expert but it would appear what you made is new, but I'm not sure if people do/can do analysis on the convergence of a NN with this operator? Sep 6 at 21:30