This is an idea which I've had sitting on my desk for a while, but hadn't gotten to until now.
Suppose that there two types of set-theoretic objects, which we'll call "elements" and "antielements." For any element $a$, there is a unique antielement $b$ such that $a=b^*$ and $\{a\}\cup\{b\}=\emptyset$, and vice-versa ('$^*$' is an involutory bijection from the class of elements to the class of antielements).
The motivating principle behind this idea was that it is naturally possible to "remove" an element from the empty set if we think of it as sort of "loan." When we take an element out of the empty set we "owe" the universe that element back - similar to that old joke about how a mathematician might interpret two people walking into an empty building followed by three people walking out.
My question is this: are there any specific, non-obvious problems incurred by the addition of antielements to ZF (or NBG)?
There are of course some surface issues like "how should cardinality work with antielements?" and the definitions of "union" and "difference" have to be modified; but these aren't too daunting on their own. What I'm concerned about is any intolerable contradictions which necessarily arise from the existence of antielements - something like "if (antielements), then not (axiom of extensionality)" - which might not be immediately obvious.
I'm not the first person to suggest this idea, either. There are a few references to the concept of antielements/antisets scattered across the literature, but I haven't been able to find anything definitive (e.g. "Introduction to ZF with Antielements", etc.). If there is a fully developed set theory with antielements, I would greatly appreciate a reference.
Papers:
https://arxiv.org/pdf/0906.3120.pdf
https://arxiv.org/pdf/1701.02993.pdf
https://hal.archives-ouvertes.fr/hal-00853859/document
https://hal.archives-ouvertes.fr/hal-00946521/document
https://www.csz.com/cyber/html/negsets.pdf
Edit:
Following Noah Schweber's answer, I decided to look into the logic behind the concept of "antielements" to see where it would lead. I started by considering the declaration of a set as a generic case of comprehension for which the formula to be satisfied is "belongs to the set":
$$\forall X.X=\{x\mid x\in X\}$$
and proceeded to define the union accordingly. What I found was that if the empty set is unique, then there cannot be a non-empty set $A$ s.t. $A\cup B=\emptyset$, because it leads to the contradiction $a\in A\implies\neg(a\in A)$.
A possible way around this is to rewrite the axiom of the empty set as:
$$\exists X.\forall x.x\in X\iff x^*\in X$$