Is the class of imaginable objects which cannot exist a worthwhile thing to talk about? Clearly, in mathematics, there are certain objects which we can imagine, yet which have no real meaning, or do not exist. The real number whose square is negative, the quantity represented by ${1 \over 0}$, integer solutions to $a^3 + b^3 = c^3$, the largest prime number, the set of all sets which do not contain themselves, etc. All of these things are imaginable and can be studied and spoken of, and their properties more or less enumerated, yet none of them actually "exist", in the sense that assuming they exist creates contradictions. 
I can imagine, then, a class containing all these objects, the class of imaginable objects which cannot exist. Is there any inherent structure to this class, or is it meaningful at all to study this class? Is this class a member of itself, and is this a paradox? And would this class be a member of the class of all mathematical objects, or the Universal set, or can this not be?
 A: The type of examples you cited are good illustrations of pre-mathematical intuitions that eventually lead to important mathematical breakthoughs, once suitable formalisations are developed.
(1) "the real number whose square is negative": this query of course led to imaginary numbers already in the 17th century, when the solution of certain cubic equations depended critically on being allowed to pass via what is today called the complex domain.
(2) "the quantity represented by $\frac{1}{0}$": leads to infinitesimals $\alpha$ and infinite numbers of the form $\frac{1}{\alpha}$.
(3) "integer solutions to Fermat's cubic equation": this type of arithmetic question has stimulated lots of research resulting in powerful tools in number theory and arithmetic.
(4) "the largest prime number": I am not sure if there is any formalisation relying on such an idea, though of course this leads to Euclid's famous result.
(5) "the set of all sets that do not contain themselves": this led to well-known paradoxes in set theory which eventually led to a widely accepted foundation to all (well, almost) of modern mathematics known as the Zermelo-Fraenkel set theory.
So I would say that the fact that these things can be imagined even though initially they don't make sense are actually the driving force behind much creative mathematics.
A: I think you have some work to do before this is really an answerable question, but here's some remarks which hopefully you find interesting.


*

*First off, in many contexts existence is relative to a theory. E.g. $ACA_0$ proves that there is no infinite binary tree without an infinite path - but such an object may exist in the much weaker theory $RCA_0$. So we can study nonexistent objects from the point of view of theories too weak to disprove their existence; in a way, this is a central theme of reverse mathematics.

*We may also play the same game with respect to the underlying logic - e.g. classical, intuitionistic, etc. This can be taken in two main directions: weakening the underlying logic to avoid contradiction (e.g reverse constructive mathematics), or embracing contradiction and weakening the underlying logic to avoid explosion (e.g. paraconsistent set theory - see Libert and Priest, the latter behind a paywall sadly). Philosophically, if you're interested in paraconsistent logic and set theory you should look at Meinong's treatment of nonexistent entities, and dialetheism in general - but I'm not at all an expert on this side of things, so I only mention them for completeness.

*Finally, going classical for a moment, there are contexts where it makes sense to talk about degrees of nonexistence. For example, consider set theory. Very roughly speaking, the class of methods called forcing take a countable transitive model of $ZFC$ and enlarge it to get a new countable transitive model of $ZFC$. However, statements outside the $ZFC$ axioms can be changed by forcing: e.g. any $V\models ZFC$ can be extended to either a $V_0$ satisfying the Continuum Hypothesis or a $V_1$ not satisfying the Continuum Hypothesis. Thus in some sense, even assuming $CH$ a counterexample to $CH$ is "less nonexistent" than a counterexample to $AC$. This sounds a bit silly, but gets much more interesting when we look at restricted classes of forcings: for example, a surjective map $\omega\rightarrow\omega_1$ cannot be added by any proper forcing, whereas a counterexample to $CH$ can be added by a proper forcing. There is a connection here with the modal logics of forcing, which have been studied by Joel David Hamkins, Benedikt Lowe, and others.

*EDIT: Yet another "classical" take on the question, inspired by $\sqrt{2}$ and $i$ - how hard is it to 'adjoin' the thing that doesn't exist? There's many ways to ask this; let me state two extremes:


*

*We fix a variety $V$ (in the sense of universal algebra) - say, the variety of groups. We also fix some specific algebra $A$ in this variety - say, the group $GL_2(\mathbb{R})$. Now, given any pair $E_0, E_1$ of equations with parameters from this group, say $E_0$ is satisfiable relative to $E_1$ if - whenever $f: A\rightarrow B$ is a homomorphism to some $B\in V$ - if $E_1$ has a solution in $B$ (with parameters moved by $f$), then $E_0$ has a solution in $B$ (with parameters moved by $f$). I don't know whether the resulting partial orders have been studied before, but they are perfectly meaningful - and maybe even interesting! (This leads to the notion, e.g., of algebraically closed groups.)

*Or look at a first-order theory $T$, with a monster model $\mathbb{M}$. We can similarly compare types over "small" parameter sets $P\subset\mathbb{M}$: say one type $q$ is realizable relative to another type $p$ if every elementary substructure of $\mathbb{M}$ containing $P$ which realizes $p$ also realizes $q$.
