# Approximate $|x|$ with a smooth function

I am trying to get the derivative of $|x|$, and I want that derivative function, say $g(x)$, to be a function of x.

So it really needs the |x| to be smooth (ex. $x^2$); I am wondering what is the best way to approximate |x| with something smooth?

I propose $\sqrt{x^2 + \epsilon}$, where $\epsilon = 10^{-10}$, but there should be something better? Perhaps Taylor expansion?

I want to use $|x|$ as part of an object function $J(x)$ which I want to minimize. So it would be nice to approximate $|x|$ with some smooth function so that I can get the analytic form of the first-order derivative of $J(x)$.

Thanks a lot.

• @Thomas But "in $x = 0$ it is not defined" is exactly what Ono wants to fix by approximating $|x|$. That is what this question is about. It's not "how to differentiate $|x|$?", it's "How to 'fix' the non-differentiability of $|x|$ at $x = 0$ by approximation?" Mar 26 '14 at 19:53
• @Thomas Yes, it can. You "smooth out" the corner. The function $\sqrt{x^2 + \epsilon}$ is one way to do just that. Another way would be to swap a small portion of the graph around $0$ with a quarter circle, but that's not nearly as smooth if you do it naively. You can do a patching like that smoothly, though, if you're smart about it, so that everywhere outside a small neighbourhood of $0$ the functions are exactly the same, and the patched function is everywhere infinitely differentiable. Mar 26 '14 at 19:58
• @Arthur The question is about (literal citation) "a derivative of $|x|$". This does not exist. It is out of question that you can approximate continuous functions, say, uniformly, by smooth functions. If that is what is asked for, the question should specify that. It does not. Mar 26 '14 at 20:02
• In fairness, both the title and the statement 'I am wondering what is the best way to approximate |x| with something smooth?' suggest a little more than an attempt to compute the derivative. Mar 26 '14 at 20:07
• @Thomas Have you read the title to the question? Titles usually give some indication as to what is asked about. I cite litterally: "Approximate $|x|$ with a smooth function". That can be done. Let $f(x)$ be a smooth function that is zero everywhere outside some small interval $(\epsilon, \epsilon)$, tops out at $f(x) = 1$ in some even smaller interval around $1$, and monotone in between. Then $(1-f(x))|x| + f(x)h(x)$ is smooth as long as $h(x)$ is smooth inside $(-\epsilon, \epsilon)$, and it's a good approximation if $h(x)$ is say, a quarter circle or parabola "glued" to the bottom of $|x|$. Mar 26 '14 at 20:09

A bit late to the party, but you can smoothly approximate $f(x) = |x|$ by observing $\partial f/\partial x = \mbox{sgn}(x)$ for $x \neq 0$. Therefore approximating the $\mbox{sgn}(x)$ function by

$$f(x) = 2\mbox{ sigmoid}(kx)-1$$

(the $k$ being a parameter that allows you to control the smoothness), we get

$$\partial f/\partial x = 2\left (\frac{e^{kx}}{1+e^{kx}} \right ) - 1$$ $$\Rightarrow f(x) = \frac{2}{k}\log(1+e^{kx})-x-\frac{2}{k}\log(2)$$

where the constant term was chosen to ensure $f(0) = 0$.

Included are plots of the function for $x \in [-5,5]$, where $k=1$ is red, $k=10$ is blue and $k=100$ is black.

Note that if you wanted to use the more smooth $k=1$ in the interval $[-5,5]$ it may be worth applying a further linear transformation i.e. ~$5f(x)/3.6$ to ensure the values at the edge of the interval are correct.

I have used your $\sqrt{x^2+\epsilon}$ function once before, when an application I was working on called for a curve with a tight radius of curvature near $x=0$. Whether it is best for your purpose might depend on your purpose.

(Side note: The derivative of $|x|$ does not exist at $x=0$. In physics, we sometimes cheat and write ${d\over dx}|x| = \rm{sgn}(x)$, the "sign" function that gives $-1$ when $x<0$, $0$ when $x=0$, and $+1$ when $x > 0$. But only when we know that the value at $x=0$ will not get us into trouble!)

Another solution would be the function $$x \rightarrow x \frac{e^{kx}}{e^{kx} + e^{-kx}} -x\frac{e^{-kx}}{e^{kx} + e^{-kx}}$$ with higher the $$k$$, closer you will be to the absolute value function. Furthermore it is equal to 0 in 0 wich is sometimes a desirable feature (compared to the solution with the root square stated above). However you lose the convexity.

$$\forall x\neq0,\dfrac{d|x|}{dx} = \frac{x}{|x|} = \begin{cases} -1 & x<0 \\ 1 & x>0. \end{cases}$$

• This is not what @Ono is after. Mar 26 '14 at 19:53
• Upvote, simply because I think the reason for the downvote is nonsense. This is the derivative. Mar 26 '14 at 19:58
• @Arthur The OP said that he wanted the derivative of $|x|$ to be a function of $x$. That is what I did
– user122283
Mar 26 '14 at 19:59
• @SanathDevalapurkar The OP also states that by approximating $|x|$ she wants a smooth solution. That is what the question is about. I.e. what smooth approximations to the absolute value function are useful. Mar 26 '14 at 20:00
• FWIW, I think a good answer should at least refer to OP's proposal, since OP asks whether the proposal is best. Mar 26 '14 at 20:05

As you and others have mentioned, functions of the form $$\sqrt{x^2 + 4\mu^2}$$ can be a good approximation to $$\left|x\right|$$ and are standard in many optimization applications where the smoothing function meets certain bounding requirements. See, for example, the paper Optimality Conditions and a Smoothing Trust Region Newton Method for Non-Lipschitz Optimization where that function is used to develop a smoothing optimization algorithm.

Another smooth approximation that has been proposed in Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches is

$$$$\mid x\mid \ \approx \frac{1}{\alpha} \left[ \log\left(1 + \exp(-\alpha x)\right) + \log\left(1 + \exp(\alpha x) \right)\right]$$$$ where $$\alpha$$ is taken to be a large positive number.