# Questions tagged [floating-point]

Mathematical questions concerning floating point numbers, a finite approximation of the real numbers used in computing.

416 questions
Filter by
Sorted by
Tagged with
38 views

### How can I tell which is closer to 0.72 without converting the binary numbers to decimal? [closed]

If I have two binary floats: $0.10111000_2$ and $0.10111001_2$ How can I tell which is closer to 0.72 without converting the binary numbers to decimal (base 10)?
36 views

### Polynomial Curve Fit without floating point

big math dummy here hoping to get some advice. I'm working on a closed loop servo system that requires a curve fit on some feedback. The controller for this system is $16$-bit. With the help of excel ...
50 views

### The gap size of floating point

In Trefethen Bau Numerical Algebra, floating point set F is defined by $\textbf{F} = \left\{\pm(m/\beta^{t})\beta ^{e}| 1 \leq m \leq \beta^{t}, e \in \mathbb{Z} \right\}$ Equivalently, by making $m$ ...
2k views

### What's special about the number $1.000000015047466$E+$30$?

I'm a programmer by trade by I've run into a weirdly special number and need some help deciphering its significance. I was writing some machine learning code that compiles into GPU kernel code and the ...
14 views

### What is the smallest positive number represented in the floating point system F(5,3,4,4) in decimal base?

I know this is easy but I have a feeling something is not right We have the base 5. 3 is the number of significant digits and the 5 varies from -4 to 4. As the exercise asks for the smallest positive, ...
1 vote
26 views

### Math behind the floating-point 2e-15 and 5e-323 in python unittest function rassertAlmostEqual [closed]

In python, there is a function rAssertAlmostEqual to check whether two floating-point numbers are almost equal. https://github.com/python/cpython/blob/main/Lib/test/...
55 views

### Condition number for the sum of two numbers

The condition number for the sum of two numbers $a$ and $b$ is $K=\frac{|a|+|b|}{|a+b|}$. But if I have $a=10$ and $b=-10$, I know that their difference is exactly $0$, so I have in practice no error, ...
1 vote
31 views