# Questions tagged [computer-arithmetic]

For questions concerning finite precision arithmetic in computers and other related concepts.

29 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
108 views

### Mathematics of two's complement

I am trying to understand the underlying mathematics of two's complement. Googling the topic gives me a lot of articles on how to invert the digits and add one, and why computers use this system ...
143 views

445 views

### How would you reduce roundoff error in “mod” when implementing a periodic function?

I recently wrote How calculators do trigonometry where I wrote a simple program for computing $\sin(x)$. For completeness, I include a slightly modified version of the program here: ...
146 views

### Numerical error analysis

I stand before the following task and I do not know how to solve it. The input parameter $$a=10^6, b=10^6 + 10^{-2}$$ will be round internally to $$a^*, b^*$$ with $$a=a^*(1 + \epsilon_1)$$ b=b^*(...
20 views

### Resultant of multivariate polynomials

I know little about resultant. As for as I know, if two univariate polynomials $f,g$ have a common zero iff $Res(f,g)=0$, is it right? Now I see an example using resultant of two multivariate ...
36 views

### Multiplication of medium sized integers

Implementations of integer multiplication often choose different algorithms for different input sizes: for smallest numbers the school book method is preferred, for largest sizes the method based on ...
268 views

### Solving BCD using 10s complement

25-7 is the question. (0010 0101 - 0111). Change 0111 to 0011 (using 10s complement) and add it, isn't it? I've been trying to solve using 10s complement but couldn't. Please help me solve the ...
34 views

45 views

### standard notation to handle representation of a real number on a computer

Is there a standard notation to handle the effective representation of some real number $x$ on a finite machine ? I have in mind some kind of braces, but I am not sure it is appropriate. Let me try to ...
56 views

### Making a function for unbounded variables that is bounded

Ok here is what I need I want to build a formula to implement into a ranking system on my website its been like 20 years since I was in school so help me out I need like 6 variables that equal at ...
My goal is to solve for $L$ in $\frac{(2k)!}{2^kk!}{2nL - L^2 \choose 2k} = \sum_{s=0}^k{L \choose s}{n-L \choose s}s!\frac{(2k-2s)!}{2^{k-s}(k-s)!)}{L-s \choose 2k-2s}.$ I tried to use the solve ...
if we take a language, $L= \{b^{2n}a^n \mid n \gt 0\} \cup \{b^{3n}a^n \mid n \gt 0\}$ would this be the same as $L=\{b^{2n}a^n b^{3n} a^n \mid n \gt 0\}$ ?