2
$\begingroup$

I just tried to calculate on my CASIO fx-991 ES the following equation:

$7F FF FF FE_{16} \cdot 2$ and $7F FF FF FE_{16} + 7F FF FF FE_{16}$

My result was MATH ERROR.

I therefore expected I overshot the 32 bit limit of the base mode of my calculator. I concluded that if that happened in a processor, the "carry flag" would be set and continued to compare my result with given solution. I was wrong. After trying to figure out why and calculating this by hand, I realised the solution is $FF FF FF FC_{16}$. When I type this directly into my calculator it accepts the input without hesitation.

Does anyone have any idea what kind of math error I ran into if the result is clearly not too big for calculation? (By the way, if I take one digit away, it's fine with the calculation. But still, why does this happen before violating the max length restriction?)

Oh, and a final thought. This is apparently calculator specific. The windows calculator has no issue with the above input.

$\endgroup$
3
  • 3
    $\begingroup$ A wild guess. This was interpreted as a signed integer, when the range may be from $-80000000_{16}$ to $7FFFFFFF_{16}$. With the latter input the calculator was forced to take it as unsigned. $\endgroup$
    – Jyrki Lahtonen
    Jul 26, 2022 at 18:29
  • $\begingroup$ You're right. $7FFFFFFF_{16} + 1$ also produces a math error. But $FFFFFFFE_{16} + 1$ does not produce a math error. That's exactly it! Hey, can you make it a real answer so I can give you the best answer? $\endgroup$
    – TheCommoner282
    Jul 26, 2022 at 18:32
  • 2
    $\begingroup$ You say you typed in $FFFFFFFC_{16}$ and it was accepted without hesitation. If you convert it to decimal it should show $-4$, something else to be aware of when working with signed integers. $\endgroup$
    – Peter Phipps
    Jul 26, 2022 at 18:50

1 Answer 1

Reset to default
5
$\begingroup$

I don't know the inner workings of this particular calculator. But my wild guess theory is that your inputs were interpreted as a signed integers. Often the range of a signed integer is from $−80000000_{16}$ to $7FFFFFFF_{16}$, and this overflowed. With the latter input the calculator was forced to take it as unsigned.

You know your tool, and can cook up further tests that give more information :-)

$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .