Since cardinality is already defined for all "existing" sets, you need to find a new way to define sets in order to have negative cardinalities.
This is by no means a hint that it cannot be done. Some people do that, e.g. this paper.
But the question is what does the notion of "set" mean in mathematics. If it means a collection with objects in it, and the notion of cardinality is the size of the set, then negative cardinality makes no sense.
What would be the disjoint union of a set of cardinality $-1$ and a set of cardinality $1$? Is it going to be empty? If not, then cardinality no longer obey some of the basic laws we expect it to obey.
In my critique of the aforementioned paper as using the terms "cardinality" and "sets" in a too-broad meaning just to get a nice title out of the paper, John Baez retorts that indeed a few centuries ago "number" was limited compared to its modern interpretation. While he is not wrong, and certainly "set" could change with time, the difference is that "set" is far more grounded in axiomatic definitions than "number". So this would be equivalent to "natural number" changing definition, which I don't see happening.
So ask yourself, simply, what does it mean for a set to have a negative cardinality? What implications will it have on cardinal arithmetic, and all that shebang? I find that a good enough "proof" that negative, or fractional, cardinalities should not be a thing in mathematics. Others will disagree, and that's fine.