Is there a recognized concept in mathematics to number systems that have an absolute value less than zero? Number line of real numbers can be represented as a 1-dimensional line of typically horizontal axis with zero being the origin. In this kind of representation, imaginary numbers would typically reside on the vertical axis, reaching to infinity in negative and positive values from origin, complex numbers residing on the plane with a non-zero distance to both of the axis.
Tried to searchengine, but didn't find such an abstract extension to be mentioned, which would give an absolute value of number/symbol to be LESS than zero?
I don't know if it would ever be useful, but neither was imaginary numbers when they were introduced to math. Nowadays we know there is lot of applications in electronics and quantum mechanics for example.
EDIT: As it turns out in the comments, imaginary numbers were already "under order" for practical applications. Practicality is not for now a point of interest personally, just curious whether field of mathematics has a name for such concept as described. Like, say I'm writing a story, where there is a creature stating it occupies a volume less than zero, which is not a negative volume, but "under zero" or "not even zero" .. And because of Reasons, I wanted to check whether such a system has a name in known maths.
 A: I'm not going to answer your question, because I think the comments do a sufficient job in doing so. Instead, I'm going to give you some perspective to thinking about extending math concepts.
When imaginary numbers are taught in schools, they often are taught as follows:

There isn't any number that is the square root of $-1$, right? Well, what if there was? Let's call it $i$.

While I understand why teachers often explain it this way, I think it confuses students. It either makes students view math axioms as voodoo that only experts are allowed to violate, or makes them think that just saying, what if rule $x$ wasn't true. What would math be like?
Don't get me wrong: plenty of interesting mathematical results have come as a result of the second thought process. But the missing piece of context here is "usefulness".
Mathematicians rarely change the axiomatic structure of a functioning system unless there is some necessary functionality they're missing (why change something that isn't broken?), or as @DavidA.Craven points out, if they want to see how far a system can be relaxed while maintaining desirable behavior. With the imaginary numbers, the key issue was not just that $\sqrt{-1}$ didn't have a meaningful interpretation in the reals, but rather, that the reals were not algebraically complete (every polynomial has a root). Thus, by extending the reals properly, mathematicians were able to minimize the amount of axioms violated (when moving from the reals to the complex numbers, you lose a natural sense of order, and you lose the canonical choice of logarithm, among other things), while gaining the benefits of the extension, such as the fundamental theorem of algebra.
From a philosophical standpoint, what you propose is not unsound. But, from a practical perspective, it's not the way we do mathematics. We don't just look at things and say, let's break this rule for the fun of it. Rather, we look at our system, see what things it doesn't do well, and try to understand what axioms are constraining the behavior. Then, we ask, how can we extend our system to one without this axiom (or perhaps with a weaker version of it, or a modified version, etc) that hopefully will resolve the constraint.
In your example of the absolute value, there is no such motivation presented, and no reason to believe there is a way to define an absolute value definition with your constraint that still fits the intuition about absolute value. That doesn't mean there potentially isn't some good use-case for this concept (in fact, norms that can be negative are quite useful), but we definitely wouldn't call it "absolute value"
A: I think you might be interested in the concept of "pseudonorms". Take a look at the space of split-complex numbers, on which the pseudonorm (the analogue of the "modulus" or "absolute value") can take negative values: https://en.wikipedia.org/wiki/Split-complex_number
A: A concept that comes close to what you describe in the question pre-edit (although I think the edit is the best part) is that of a negative definite quadratic form.
The generalization of the absolute value on the one-dimensional line to higher dimensional spaces is a norm. To compute the norm of a point you compute the squares of all the coordinates and add them. Now roughly speaking a quadratic form is a generalization of this concept where you take the squares of coordinates and instead of ordinarily adding them you first multiply each of them with a pre-chosen number (positive or negative) and then add the outcomes.
Quadratic forms are of interest to a wide range of mathematical topics, and hence are studied 'on their own'. If you want to, from that perspective, recognize the norms among the more general zoo of quadratic forms, the crucial property that the norms have is that they take only non-negative values (with zero occurring only if started with feeding the point zero into the norm). This property that some quadratic forms have and others don't is called being positive definite.
Naturally there is a mirror symmetric property called being negative definite meaning taking only non-positive values, with zero occurring only if you use the zero vector as input. In some contexts the negative definite quadratic forms are more natural/interesting than the positive ones. Among all quadratic forms that are not norms they are the ones closest to norms I would say. And as I said: norms are the natural extension of absolute values to higher dimensions.
That said: quadratic forms don't have the interpretation of a volume, as computing volumes involves multiplying 3 numbers together rather than just two. So the post-edit version of your question might need a different answer.
A: This quote

I'm writing a story, where there is a creature stating it occupies a
volume less than zero, which is not a negative volume, but "under
zero" or "not even zero"

from your question suggests that you might be better off asking on the world building stackexchange. Start with the implications this sort of "negative volume" has on the plot of your story. If not much then you don't need to justify it by a reference to any real mathematics. Anything reasonably convincing at first glance will do.
It might suffice to say it has "infinitesimal volume". That would be greater than $0$ but less than any ordinary number. In the part of mathematics called "nonstandard analysis" there are infinitesimal numbers. You could also consider the surreal numbers.
A: In the context of complex numbers and use of the Euler's formula, once we encounter a negative number (argument is $\pi/2$) with negative modulus, we automatically assume it is equal to a positive number (argument $0$) with positive modulus. Similarly, when we encounter a positive number (argument $0$) with negative modulus, we automatically bring it to the conventional form of a negative number with positive modulus.
In other words, whereever we encounter a number with negative modulus, we change the argument by $\pi/2$ and make modulus positive so to bring the number to its canonical form.
Thus we can bring any complex number to a form where it has positive modulus.
On the other hand, if you consider split-complex numbers and modulus defined as $|a+bj|=\sqrt{a^2-b^2}$ (some authors define "modulus" differently), you will have some numbers with imaginary (in the usual sense) modulus. For instance, the split-complex unity $j$ has modulus $|j|=i$, the usual imaginary unit. Since split-complex numbers and complex numbers are often combined to form tessarines, this is not a big problem.
That said, there are numbers with imaginary modulus in some sense, but if you encounter a negative modulus, you change the sign of the number and consider the modulus positive.
