Questions tagged [rounding-error]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

What is the name of the quantization method where you truncate before adding a half?

Many years ago, when designing a fixed-point CORDIC algorithm in hardware, I stumbled across a method of quantization by which you could "round" by truncating to the desired precision before ...
mattgately's user avatar
0 votes
1 answer
30 views

How to stop a negative exponential from rounding to zero?

I'm doing materials homework and calculating vacancy density, which has some large constants (Na and k), and have to do the function. the right side of the density equation is exp(-Q/(kT)), which ...
Erwin Davinky's user avatar
0 votes
0 answers
41 views

Why is this the expression for the rounding error in the three-point midpoint formula for the approximation of the first derivative?

so I understand that in using the three-point midpoint formula, $$f^{(1)}(x)= \frac{y_{+1}-y_{-1}}{2h} + \frac{e_{+1}-e_{-1}}{2h} - \frac{h^2}{6} f^{(3)}(\xi^*), \ \ \ \ \ \xi^* \in [a,b]$$ round-off ...
trxxns's user avatar
  • 1
2 votes
0 answers
79 views

Prime twin counting by $\pi_2(t^2) =^? \sum_{2<j<t^2} (-2)^{\omega(j)} (1/2)(\lfloor{\frac{t^2}{j}}\rfloor +\lfloor{\frac{t^2-2}{j}}\rfloor) +C$?

Let $\omega(n)$ count the number of distinct prime factors of the integer $ n \geq 2$. This $\omega(n)$ is called the prime omega function. Inspired by these ideas : Improved sieve for primes and ...
mick's user avatar
  • 15.9k
0 votes
1 answer
26 views

Relative error of $\mathrm{z_0}$ with respect to z

Suppose that for a problem with solution $\mathrm{z}$, our approximated result was $\mathrm{z_0=175.002043}$ and the relative error committed $\mathrm{\frac{|\Delta z_0|}{|z|}} \le \mathrm{5x10^{-4}}$....
J P's user avatar
  • 333
0 votes
0 answers
38 views

Isolate variable when rounding

My goal is to isolate frame from this equation: $$ ms = round(frame * {1\over fps} * 1000) $$ $$ ms \in \mathbb{N} $$ $$ frame \in \mathbb{N} $$ $$ fps \in \mathbb{R+} $$ Note: $ round(0.5) = 1 $ ...
jeremie bergeron's user avatar
4 votes
3 answers
562 views

Why does deg(arcsin(sin(rad(degree)))) not produce degree?

I have an angle in degrees and I want it to be encoded between $-1, ..., 1$ to make it easier for using in a neural network. I thought it might be a good idea to first convert the angle into radians ...
binaryBigInt's user avatar
3 votes
0 answers
109 views

Justification for the definition of relative error, why is it not a metric?

The absolute error and relative error operators are very commonly encountered while reading about topics from the fields of floating-point arithmetics or approximation theory. Absolute error is ${ae(a,...
user2373145's user avatar
0 votes
0 answers
22 views

A rounding and distribution problem

This is a real-life problem I'm trying to solve, I have an invoice with three items A, B, and C on it, and the total cost of this invoice is $4.98, below is how much each item costs A $1.99 B $2.17 C $...
Arch1tect's user avatar
  • 139
0 votes
0 answers
57 views

How to determine the precision of intermediate results in a calculation process?

How many digits of precision should be retained in the calculation process if I want the result of a mathematical operation to have n digits of precision? For example, if I want the result of ...
guan boshen's user avatar
1 vote
0 answers
31 views

Is there a name for such rounding algorithm?

When I compute a number where an approximation is needed but no accuracy is specified, I usually scan the first few digits and if I find a zero, then I truncate the decimals before the first zero. For ...
Zuriel's user avatar
  • 5,311
-1 votes
1 answer
64 views

Which rounding off method to follow? [duplicate]

In a book named " Concepts of Physics" by H.C Verma , it was said that " when there is 5 after the decimal point and the number immediately before the decimal point is an even number it ...
World Producer's user avatar
0 votes
0 answers
38 views

Matlab/Octav, minimising rounding error during computation

My question regards best practices to minimise rounding errors in computer calculations. In my Numerical Methods course I have been taught that the size of memory (presumably RAM) Matlab uses during ...
Erik Eriksson's user avatar
0 votes
0 answers
59 views

Error bounds in numerical methods

I'm reading Sauers Numerical Analysis. There is an exercis where I have to use the bisect method to calculate a height within $\pm 1$mm. (1.1 computer problems nr 9.) I use bisect method with a while ...
Erik Eriksson's user avatar
0 votes
1 answer
108 views

Total error for the finite difference approximation

I was reading Scientific Computing, An Introductory Survey, by Michael Heath. In the Example 1.11, he made a Finite Difference Aproximation, with the usual approximation : $f’(x)\neq \frac{f(x+h)-f(x)}...
RES's user avatar
  • 13
1 vote
0 answers
43 views

Algorithmic error computing $\frac{e^x -1}{x}$

It is well know that in order to calculate the algorithmic error of a function one can use backward analysis with using the visual representation of a graph with nodes the $i-$th step of the algorithm ...
jacopoburelli's user avatar
1 vote
1 answer
134 views

Does using smaller floating-point numbers decrease rounding errors?

I started learning about floating point by reading "What Every Computer Scientist Should know About Floating-Point Arithmetic" by David Goldberg. On page 4 he presents a proof for the ...
Thanks for flying Vim's user avatar
2 votes
0 answers
34 views

correct number of decimals

I'm reading Sauers textbook on Numerical Methods. An exercise where I use the Bisection method to apporoximate a root to a given function, the answer is supposed to be given with six correct decimals. ...
Erik Eriksson's user avatar
0 votes
1 answer
60 views

Representation of rounding error in floating point arithmetic. [duplicate]

It is well known that in a Floating point number system: $$ \mathbb{F}:=\{\pm \beta^{e}(\frac{d_1}{\beta}+\dots +\frac{d_t}{\beta^t}): d_i \in \{0,\dots,\beta-1\},d_1\neq 0, e_{\min}\leq e \leq e_{\...
Henry T.'s user avatar
  • 1,287
1 vote
0 answers
30 views

Expression of sum in floating point system

This is a question of an exam on Numerical Analysis I had: Consider the floating point system of base $2$, maximum number of decimals $53$, maximum exponent $1025$ and minimum exponent $-1022$. That ...
Little Jonny's user avatar
3 votes
0 answers
228 views

Numerical analysis: Find binary representation of $\pi$ and $e$

I am studying Numerical Analysis. In the beginning of the book they go through a method of converting base 10 decimals to base $2$. Multiply the decimal number $a<1$ with $2$, the whole part ($0$ ...
Erik Eriksson's user avatar
2 votes
0 answers
80 views

Numerically stable evaluation of factored univariate real polynomial

Suppose we have a real univariate factored polynomial, meaning we have its factors: an arbitrary number of polynomials of degree less than or equal to two. To simplify things, if necessary, let's ...
user2373145's user avatar
1 vote
1 answer
142 views

Von Neumann stability for inhomogeneous PDE

I've got an inhomogeneous PDE of the following form: $$\alpha\partial^2_xu+\partial_tu=f$$ with $\alpha<0$ and a source term $f$. I descretise $u$ according to $u_{m,n}=u(m\Delta t,n\Delta x)$ ($f$ ...
thorr's user avatar
  • 23
0 votes
0 answers
65 views

rounding error of implicit and explicit method

To solve the differential equation $y'=f(t,y), y(0)=y_0$ consider a explicit $y_{n+1}=y_n+hf(t_n,y_n)$ and a implicit $y_{n+1}=y_n+hf(t_{n+1},y_{n+1})$ method to solve the ODE. If in the explicit ...
Robert's user avatar
  • 345
0 votes
1 answer
1k views

Absolute error, Relative error, Percentage error computation

Round off the numbers $865250$ to four significant figures and compute Absolute Error, Relative Error, and Percentage error My calculation is Absolute error-$50$, Relative error -$0.000057$, ...
Mark Taylor's user avatar
0 votes
0 answers
57 views

Is there a "standard" method of breaking ties when rounding in the field of mathematics?

I recently found that software packages deal with ties in different ways: round ties away from zero: round(0.5) = 1.0, ...
Saaru Lindestøkke's user avatar
-2 votes
1 answer
25 views

Simulation of difference equation

I am struggling with the following question: $x_{n+2} - \frac43 x_{n+1} + \frac13 x_n = 1$, $n\ge0$, $x_0 = -\frac32$, $x_1 = -1$ (i) What happens when you simulate (i) with $64$-bit floating numbers ...
Boshi's user avatar
  • 39
3 votes
2 answers
147 views

Explanation for MATLAB floating point number calculation?

I am a beginner studying scientific computation, more specifically floating point numbers and precision in matlab. When testing the outputs of 2 of the following equations, I am not sure how matlab ...
cronk's user avatar
  • 31
-1 votes
1 answer
45 views

Why do equivalent trig functions differ in values they yield? [closed]

Suppose we are given [1-cos(x)]/sin(x). Through calculations, we know that this will be equal to sin(x)/[1+cos(x)]. Now, if we assume an angle value, say x=0.006, they will yield 5.23598532510^-5 ...
thepajama's user avatar
0 votes
0 answers
72 views

Find the number of true significant digits in the function $ z=\frac{\sin (3 x+0.7 y)}{\mathrm{e}^{0.4 x+0.5 y}} \cdot(7 x+\ln y) $

I just start learning about error theory, rounding etc. Concerning the solution, we need to find the absolute error of the function $\Delta_{z^*}$ and then use it to find the number of true ...
Maroon Racoon's user avatar
2 votes
0 answers
25 views

Cases when rounding makes approximations exact

This is a bit of a soft question, but I hope it's still precise enough. I'm interested in finding cases where a complicated expression has a simple approximation which becomes exact again upon ...
J_P's user avatar
  • 2,138
0 votes
1 answer
76 views

Summing series - Error propagation

Why sum down and sum up given different results? I know that this has a relation with error propagation, but I can't figure why. ( You can use whatever programming language you want, if you choose a ...
Vithor's user avatar
  • 96
0 votes
1 answer
33 views

Where to truncate original figures when the output would be rounded?

Is there any rule to determine how many decimal places a rational and integer number placed into the abstract, binary function of multiplication be truncated to if the output should be rounded to a ...
duckegg's user avatar
  • 669
2 votes
0 answers
42 views

Computing log of a number with low precision calculator

Say I have a low-precision calculator that displays numbers with only 8-digits mantissa. For example, 0.00432137378 would read ...
fricadelle's user avatar
2 votes
0 answers
47 views

Truncation error - subtraction

i try to eliminate truncation error with subtraction in calculating root of function where$$x_1=\frac{-b-\sqrt{b^2-4c}}{2},\quad b<0,\quad 0<c\ll 1.$$Does someone have an idea of how to change ...
carl_799's user avatar
1 vote
1 answer
373 views

Rounding error of matrix multiplication when one of the matrices is orthogonal

I am studying Scientific computing from Biswa Nath Datta's Numerical Linear Algebra and Applications and there is a corollary after explaining matrix multiplication rounding error described below. if $...
asha soroushpoor's user avatar
-4 votes
1 answer
79 views

Finding roots of the equation $x^2−40x+1=0$ [closed]

The problem: Given the equation $x^2−40x+1=0$, find its roots to five significant digits. Use $√399≐19.975$, correctly rounded to five digits. Can anyone help me solve this problem? My thoughts: ...
Dilip Kumar Jena's user avatar
0 votes
1 answer
68 views

$ \left\lfloor 10^{\lfloor n \rfloor} \pi \bmod 10 \right\rfloor $ - does this function give the nth decimal place of pi?

Function to round the nth decimal place of pi to the nearest integer. For example, for pi, n = 0, y = 3. n = 1, y = 1. n = 2, y = 4. And so on and so forth. Gives me good results until n = 17, which ...
Josh's user avatar
  • 467
0 votes
2 answers
102 views

Google rounds wrongly?

I know, this is not the Google bug tracker, but maybe the problem is my poor understanding of math. Ask Google to convert pica to mm. (Pica is a measurement unit ...
john c. j.'s user avatar
0 votes
1 answer
49 views

44.1 = 40.0? Need help with rules of addition in Significance arithmetic

The Wikipedia page for Significance arithmetic uses this example in the rules for addition: 9.9 9.9 9.9 9.9 3.3 + 1.1 ---- 40.0 Each addend has two ...
jnthn's user avatar
  • 376
0 votes
0 answers
108 views

relative condition number square root of 2

I need some help with the following task: I should calculate the term $f_1=(\sqrt{2}-1)^6$ therefore I should approximate $\sqrt{2}$ with $1.4$. I have some other alternative terms $f_2=(\sqrt{2}+1)^...
Walter Frosch's user avatar
0 votes
0 answers
288 views

No. of significant figures in absolute value w.r.t true value and relative percentage error

To find the no. of a significant figure in absolute value $= 0.05411$ with respect to true value $= 0.05418$ and the relative percentage error. Here's my solution: But I ain't sure whether it's ...
Archana Dash's user avatar
0 votes
0 answers
31 views

How would you design an algorithm to solve this decision problem?

Suppose I want to design an algorithm that, for an arbitrary polynomial $p$, returns YES iff there are two roots $z_1$ and $z_2$ of $p$ such that $\left|z_1 - z_2\right| = 1$. How do I design such an ...
matty_k_walrus's user avatar
0 votes
0 answers
72 views

How precise is the result after calculating a/b?

I have two numbers, $a$ and $b$, where the precision of $a$ is $n$ bits and the precision of $b$ is $m$ bits. How many bits of precision are preserved after calculating a/b on a normal computer ...
Gamer2015's user avatar
  • 121
0 votes
2 answers
95 views

3D rotating using whole number increments.

I am working on some 3D rotation calculations but have some unusual limitations. I have to move the coordinates relative to the current coordinates ex: ...
patrickbies's user avatar
0 votes
1 answer
579 views

How to *correctly* round to the nearest multiple

I am struggling to find the correct implementation for rounding to the nearest multiple. I thought the simple arithmetic of [number/multiple]*multiple would give me ...
S1r-Lanzelot's user avatar
3 votes
2 answers
4k views

Total average of averages not same as the average of total values

I am having a strange problem while computing the overall percentage. Let me demonstrate my problem using an example. Assume that I am receiving multiple batches of apple from the vendor and the ...
Chris Aung's user avatar
0 votes
0 answers
98 views

How do I find the maximum error that could result from early rounding?

I'm looking at code that does the following: ...
James's user avatar
  • 119
1 vote
2 answers
72 views

How does truncating a series affect "upstream" values in the series?

It is known that truncating the Gregory’s series to 5,000,000 terms leads to an "almost but not quite" value for π: $$ \pi=4 \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{2 k-1}=4(1-1 / 3+1 / 5-1 / ...
Cybernetic's user avatar
2 votes
1 answer
74 views

Absolute error formula

Trying to figure out a problem in my textbook. In my exercise in a textbook, a problem says that Determine the error in the approximation given that the actual length is $3.7437137$. And in the ...
JayEstrera's user avatar