# Questions tagged [floating-point]

Mathematical questions concerning floating point numbers, a finite approximation of the real numbers used in computing.

449 questions
Filter by
Sorted by
Tagged with
19 views

### Floating Point Precision Algorithm

In my database, data stored as a precision of 10 digits Decimal(30,10). User can enter x or 1/x. I need to save in 1/x. If user enters ...
• 471
29 views

• 1,539
15 views

### How to Multiply 2 arrays with unique non-integers to prodice an array with unique results?

Is there an Algortihm/formulae to multiply two arrays (1D & 2D) of unique numbers such that the resultant array contains unique results. Would one have to create the 2 initial arrays in a certain ...
1 vote
55 views

### What is the set of all numbers that can be represented with a floating-point format?

Computers use single- (or, for more precise calculations, double-) precision floating-point formats to represent a subset of real numbers. While a decent chunk of real numbers can be stored with these ...
• 193
23 views

• 2,006
84 views

### What's the term for the coefficient that change any floating point number to its next or previous value?

I calculated that the following two coefficients will reduce any finite floating point number to its next exact lower value: Single precision: 0.99999994 Double ...
• 651
58 views

### Computational Game Theory, Why aren't Nash Equilibrium calculated with rational numbers instead of floating point numbers?

In algorithms like Counterfactual Regret Minimization/Regret Matching, most implementations are coded using floating point numbers to represent (average) strategies, and the value of the game is ...
276 views

### Is There Any Good Way of Checking Floats for Approximate Equality?

I'm an EE, not a mathematician, so I apologize in advance if this question is dumb. When working with floating point numbers, it is typically a bad idea to expect bitwise equivalence for reasons that ...
47 views

### Find the smallest/largest scalar that converts a vector of reals to a vector of integers

Suppose there are two vectors: $\mathbf{x} \in \mathbb{R}^N$ and $\mathbf{y} \in \mathbb{Z}^N_+$. If I know that $\mathbf{x} = \alpha \cdot \mathbf{y}$, where $\alpha \in \mathbb{R}$, is there a way ...
• 3,262
102 views

### How to do backward error analysis of $Ax=b$?

I know the definition of forward error, backward error and condition number. The following is a backward error analysis I think. Let $A$ be a square matrix of order $n$ and $\hat{x}$ be a approximate ... 96 views

### Polynomial Curve Fit without floating point

big math dummy here hoping to get some advice. I'm working on a closed loop servo system that requires a curve fit on some feedback. The controller for this system is $16$-bit. With the help of excel ...
1 vote
97 views

### The gap size of floating point

In Trefethen Bau Numerical Algebra, floating point set F is defined by $\textbf{F} = \left\{\pm(m/\beta^{t})\beta ^{e}| 1 \leq m \leq \beta^{t}, e \in \mathbb{Z} \right\}$ Equivalently, by making $m$ ...
• 23
2k views

### What's special about the number $1.000000015047466$E+$30$?

I'm a programmer by trade by I've run into a weirdly special number and need some help deciphering its significance. I was writing some machine learning code that compiles into GPU kernel code and the ...
• 223
97 views

### What is the smallest positive number represented in the floating point system F(5,3,4,4) in decimal base?

I know this is easy but I have a feeling something is not right We have the base 5. 3 is the number of significant digits and the 5 varies from -4 to 4. As the exercise asks for the smallest positive, ...
• 27
94 views

### Condition number for the sum of two numbers

The condition number for the sum of two numbers $a$ and $b$ is $K=\frac{|a|+|b|}{|a+b|}$. But if I have $a=10$ and $b=-10$, I know that their difference is exactly $0$, so I have in practice no error, ...
• 185
1 vote
179 views

• 23
1 vote
265 views

### Number of FLOPs for the inner and outer vector product?

I'm trying to educate myself on the relative cost of the cross and dot products relative to the number of floating point operations (FLOPs) each one requires. My understanding based on this paper (see ...
39 views

• 151