What is the symbol for a a result that overflows? Suppose we have an unsigned $8$ bit number (min=$0$, max=$255$).
the result of "$200 + 200$" overflows to $144$
the result of "$100 - 200$" (under?)overflows to $156$
Is there are mathematical symbol to represent this?
 A: You can denote it using $x \pmod {y+1}$ as long as your minimum is $0$ and the maximum value is $y$.
Example $1$:
$$200 + 200 = 400 = 256 + 144$$
$$200 + 200 \equiv 144 \pmod {256}$$
Example $2$:
$$100 - 200 = - 100 = -256 + 156$$
$$100 - 200 \equiv -100 \pmod {256}$$
A: "Overflow" is the wrong term here:

to flow over the edge or brim of (a receptacle, container, etc.).

ref: http://www.dictionary.com/browse/overflow
In the cited example, you have an integer that is too large to fit in the data format you are using to represent an integer: specifically 8-bit binary numerals.
The operation you are describing is not overflow, but a common reaction to overflow that a lot of computer environments practice: to store just the least significant 8 bits of the (two's complement) result.
In computer lingo this is sometimes called "wrapping", envisioning the number line being wrapped up around a circle whose circumference has the 256 possible values you can store.
Mathematically, this turns out to be the same as computing a reduced representative modulo 256. Or more accurately, you aren't performing addition of integers, but you are performing addition modulo 256.
To specify that the operation and the result are equivalent modulo 256, one writes
$$ 200 + 200 = 144 \pmod{256} $$
Sometimes, people use the remainder operator (which, confusingly, is often notated as $\bmod$), and would write
$$ (200 + 200) \bmod 256 = 144 $$
Some standards even explicitly invoke this mathematical operation — + on 8-bit types isn't defined to be addition of integers, but addition modulo 256.

Incidentally, some computing environments also practice other behaviors; e.g. you can often ask for saturating arithmetic, under which $200 + 200$ would simply produce the maximum value $255$. The C and C++ standards decree that overflow (of signed integer types) shall not happen; if you write a program where it does, undefined behavior results meaning the standard allows the program to do anything at all.
