# Questions tagged [rounding-error]

The tag has no usage guidance.

177 questions
Filter by
Sorted by
Tagged with
25 views

### What is the name of the quantization method where you truncate before adding a half?

Many years ago, when designing a fixed-point CORDIC algorithm in hardware, I stumbled across a method of quantization by which you could "round" by truncating to the desired precision before ...
• 103
30 views

### How to stop a negative exponential from rounding to zero?

I'm doing materials homework and calculating vacancy density, which has some large constants (Na and k), and have to do the function. the right side of the density equation is exp(-Q/(kT)), which ...
41 views

### Why is this the expression for the rounding error in the three-point midpoint formula for the approximation of the first derivative?

so I understand that in using the three-point midpoint formula, $$f^{(1)}(x)= \frac{y_{+1}-y_{-1}}{2h} + \frac{e_{+1}-e_{-1}}{2h} - \frac{h^2}{6} f^{(3)}(\xi^*), \ \ \ \ \ \xi^* \in [a,b]$$ round-off ...
79 views

### Prime twin counting by $\pi_2(t^2) =^? \sum_{2<j<t^2} (-2)^{\omega(j)} (1/2)(\lfloor{\frac{t^2}{j}}\rfloor +\lfloor{\frac{t^2-2}{j}}\rfloor) +C$?

Let $\omega(n)$ count the number of distinct prime factors of the integer $n \geq 2$. This $\omega(n)$ is called the prime omega function. Inspired by these ideas : Improved sieve for primes and ...
• 15.9k
26 views

### Relative error of $\mathrm{z_0}$ with respect to z

Suppose that for a problem with solution $\mathrm{z}$, our approximated result was $\mathrm{z_0=175.002043}$ and the relative error committed $\mathrm{\frac{|\Delta z_0|}{|z|}} \le \mathrm{5x10^{-4}}$....
• 333
38 views

### Isolate variable when rounding

My goal is to isolate frame from this equation: $$ms = round(frame * {1\over fps} * 1000)$$ $$ms \in \mathbb{N}$$ $$frame \in \mathbb{N}$$ $$fps \in \mathbb{R+}$$ Note: $round(0.5) = 1$ ...
562 views

### Why does deg(arcsin(sin(rad(degree)))) not produce degree?

I have an angle in degrees and I want it to be encoded between $-1, ..., 1$ to make it easier for using in a neural network. I thought it might be a good idea to first convert the angle into radians ...
• 404
109 views

• 13
1 vote
43 views

### Algorithmic error computing $\frac{e^x -1}{x}$

It is well know that in order to calculate the algorithmic error of a function one can use backward analysis with using the visual representation of a graph with nodes the $i-$th step of the algorithm ...
• 5,544
1 vote
134 views

### Does using smaller floating-point numbers decrease rounding errors?

I started learning about floating point by reading "What Every Computer Scientist Should know About Floating-Point Arithmetic" by David Goldberg. On page 4 he presents a proof for the ...
34 views

### correct number of decimals

I'm reading Sauers textbook on Numerical Methods. An exercise where I use the Bisection method to apporoximate a root to a given function, the answer is supposed to be given with six correct decimals. ...
60 views

• 300
74 views

### Absolute error formula

Trying to figure out a problem in my textbook. In my exercise in a textbook, a problem says that Determine the error in the approximation given that the actual length is $3.7437137$. And in the ...