1
$\begingroup$

Can differentiation be defined for floating point numbers (e.g. 32 bit floats or 64 bits doubles)?

I think one can have limit points in floating point numbers, I read somewhere that they have a discrete topology, and discrete topologies can be made coarser, and then limit points exist.

However when I tried using the c++ method for the closest bigger floating point number starting from zero the values did not seem to be stable. It usually goes like : (NaN|Inf|0|explosions)....(0|explosions)...(close to zero| truncated/rounded values)...(wrong value, diffing by a small value)...(just plain wrong value|explosions) – So i did not manage to create a cauchy sequence for the difference formula I used.

An approach I saw was using interval arithmetic, perhaps that is a suitable "coarser topology".

Can one define $\mathcal{C}^n$ differentiability over the floating point numbers?

$\endgroup$
5
  • $\begingroup$ A simpler question with essentially the same structure: can you define differentiation for functions with domain $\Bbb Z$? $\endgroup$
    – Mario Carneiro
    Jun 27, 2018 at 22:02
  • $\begingroup$ Well, the set has fewer elements than the integers, but it is probably similar. Did not manage to find any question similar enough to mine. $\endgroup$
    – Emil
    Jun 27, 2018 at 22:08
  • 1
    $\begingroup$ There was an answer in this post addressing a related subject: math.stackexchange.com/questions/2213240/… You might be interested in reading it. $\endgroup$
    – zwim
    Jun 27, 2018 at 22:55
  • $\begingroup$ @zwim: thank you, I must try float_next(x) and x next time I go at it. $\endgroup$
    – Emil
    Jun 28, 2018 at 7:13
  • 1
    $\begingroup$ See also math.stackexchange.com/a/2019715/115115, math.stackexchange.com/a/2488893/115115 on why minimal step sizes do not work. There is floating point noise in the function evaluation of size $\sim\mu/h$ and the method error $O(h^p)$. The cumulative error is minimal where these terms balance, at about $h\sim \mu^{1/(p+1)}$. $\endgroup$
    – Lutz Lehmann
    Jun 28, 2018 at 16:16

1 Answer 1

Reset to default
4
$\begingroup$

A simpler question is: how can you define the derivative of a function $f$ on the integers? Since differentiation is a local property, you can restrict your attention to finite intervals of integers like $\{0,...,n\}$ if you prefer.

There's an immediate problem: since differentiation is a local property, it should only depend on values sufficiently close to the input value $a$, for any definition of "sufficiently close", but on the integers (or the floating point numbers), as a direct consequence of discreteness, you can zoom in enough that there are no points sufficiently close to $a$ other than $a$ itself. So any line that passes through $f(a)$ is a perfect approximation of the function $f$ in this neighborhood, and so the derivative is degenerate - every number is a derivative of $f$ at $a$.

There are ways to circumvent this problem, but they all assume that there is some underlying continuous function that is being approximated by $f$. The function $f$ itself just doesn't have enough information in it to have a well defined derivative.

For example, if we "connect the dots" between $f(a-1),f(a),f(a+1)$ by a quadratic function, essentially making the best possible use of the directly neighboring information, we obtain the "central difference" numerical scheme for approximating the derivative at $a$, $$f'(a)\approx\frac12 (f(a+1)-f(a-1)).$$ In general, we often consider this as a function of the step size $h\in\Bbb R^+$, i.e. considering functions defined on $h\Bbb Z$ instead of just $\Bbb Z$. In this case we can say $$f'(a)=\frac{f(a+h)-f(a-h)}{2h}+O(h^2)$$ whenever $f$ is a $C^2$ real function. Notice that the error bounds depend on there existing a smooth real function $f$ already, but the right hand side (without the error bound) can be evaluated on grid points, which is the function you actually have available.

This is just the beginning of the field of numerical analysis, which often uses techniques like floating point computation and tries to relate this to idealized mathematical functions and derivatives.

$\endgroup$
6
  • $\begingroup$ Unfortunately I have already studied numerical analysis. When using e.g. a central difference formula one sees that the first thousands of "next bigger" floating point values are riddled with NaN and Inf, after many thousand more it seems to hover around a value within some bounds, and then i explodes to big values (at least for the function I was computing all possible central differences for). Perhaps some interpolation/(extrapolation?) on all the calculated central differences before they "explode" would be a nice approach to get as many digits as possible, "explode" is hard to quantify tho $\endgroup$
    – Emil
    Jun 28, 2018 at 6:16
  • $\begingroup$ Could you expand the "all lines go through" part? Think I did not get the point. (What is a line in context of floats? Under what circumstances does the line exist? (perhaps there are no floats where the line should be?)) $\endgroup$
    – Emil
    Jun 28, 2018 at 7:24
  • $\begingroup$ As mentioned in the linked question by zwim, the best choice of difference for compromising between accuracy of approximation and roundoff error is approximately $\sqrt\varepsilon$ where $\varepsilon$ is the machine epsilon, the difference between adjacent floating point values near your target point $a$. $\endgroup$
    – Mario Carneiro
    Jun 29, 2018 at 7:18
  • $\begingroup$ Re: "all lines go through", I am speaking about the mathematics of a derivative here. There are no floating point considerations involved. Given a function which is only defined at discrete values in some set $A$ (they could be floating points or some finite set of values, or the integers), around every point $a\in A$ there is a small region containing no other points in $A$. Now consider a linear function such as $g(x) = c(x-a)+f(a)$, defined on all real numbers. ... $\endgroup$
    – Mario Carneiro
    Jun 29, 2018 at 7:23
  • $\begingroup$ ... This function is obviously differentiable with derivative $c$, and it is a perfect approximation of $f$ on the small region around $a$, since the only place where we can evaluate both $f$ and $g$ is $a$ and at that point $g(a)=f(a)$. Therefore $c$, the slope of $g$, is a derivative of $f$ at $a$ (and the same argument goes regardless of what $c\in\Bbb R$ is). This is a rather pathological case, and we usually avoid it by saying that the derivative is only defined when the domain of the function has no isolated points. A discrete set is composed entirely of isolated points. $\endgroup$
    – Mario Carneiro
    Jun 29, 2018 at 7:26

Not the answer you're looking for? Browse other questions tagged .