Why is it incorrect to say, $\lim_{x\rightarrow a}f(x)\notin\mathbb{C}$? My grade $12$ calculus teacher told me I cannot write the following:
$$\lim_{x\rightarrow a}f(x)\notin\mathbb{C}$$
to say that the limit does not exist. The only reasoning she gave was "I think you should have more experience working with complex numbers before you say that." It's been said a million times that there is no formal notation to say a limit does not exist and that it's best to just write it out or use 'D.N.E'. But I'm curious to know what is wrong with this statement mathematically. Can the limit of $f(x)$ exist and be outside $\mathbb{C}$? Thanks in advance.
 A: Short Answer
There are times when it is appropriate to write $\lim_{x\to a} f(x) \not\in X$, where $X$ is some set of interest.  However, if $f$ is a complex function, it is not possible for $\lim_{z\to a} f(z)$ to exist but not be in $\mathbb{C}$, hence the notation $\lim_{z\to a} f(z) \not\in \mathbb{C}$ is confusing and ambiguous.  I would avoid this notation.
Discussion
Depending on how, precisely, the notation $\lim_{x\to a} f(x)$ was defined, there may be nothing wrong with writing $\lim_{x\to a} f(x) \not\in \mathbb{C}$ as a sort of rough synonym of "the limit does not exist as a complex number."  This might be okay in the right context.  However I, personally, don't like this use of notation, and I think that it is likely to cause some confusion.  To explain in more detail, let's start with a basic definition:

Definition: Let $a, L \in \mathbb{C}$ and suppose that $f$ is a function which is defined on some ball centered at $a$ (though not necessarily at $a$ itself).  If for any $\varepsilon > 0$ there exists some $\delta > 0$ such that
  $$ |z - a| < \delta \implies |f(z) - L| < \varepsilon, $$
  then we say that the limit of $f(z)$ as $z$ approaches $a$ is $L$, and write
  $$ \lim_{z\to a} f(z) = L. $$

The notation is defined only in cases when the limit actually does exist.  Hence when I write $\lim_{x\to a} f(x)$, I am already assuming that this limit exists.  Of course, if for any $L \in \mathbb{C}$ I can find some $\varepsilon$ such that no $\delta > 0$ does the job required in the definition, then I can say that the limit does not exist, which I might write as
$$
\lim_{z\to a} f(z) \text{ DNE}
\qquad\text{or}\qquad
\lim_{z\to a} f(z) \text{ does not exist.}
$$
This is kind of an abuse of notation, but it is perfectly understandable in most contexts.  Since the goal of mathematical writing is clear communication, we let it stand.  Indeed, we already overload the notation a bit by considering infinite limits and limits at infinity[1], so it is entirely reasonable to use the notation $\lim_{z\to a} f(z)$ even when the limit does not exit in the sense defined above.
On the other hand, the notation
$$
\lim_{z\to a} f(z) \not\in \mathbb{C}
$$
implies something other than "the limit does not exist."  Rather, it seems to say that the limit exists, but is not a complex number.  In principle, such a statement could hold.  For example, consider the sequence of rational numbers
$$\left(
a_0 = 1,
a_1 = 1 + \frac{1}{2},
a_2 = 1 + \frac{1}{1+\frac{1}{2}},
a_3 = 1 + \frac{1}{1+\frac{1}{1+\frac{1}{2}}},
\dotsc
\right).
$$
In each term, replace the fraction $\frac{1}{2}$ with $1/(1+\frac{1}{2})$.  Each term in this sequence is rational.  However
$$ \lim_{n\to\infty} a_n = \varphi = \frac{1+\sqrt{5}}{2}, $$
which is not a rational number.  So the limit of this sequence exists, but is not a rational number.  Therefore
$$ \lim_{n\to\infty} a_n \not\in\mathbb{Q}. $$
This notation implies that the sequence has a limit, but that this limit doesn't live in the set $\mathbb{Q}$.  Similarly, if we write $\lim_{z\to a} f(z) \not\in\mathbb{C}$, this implies that the limit exists, but is not a complex number.
But this is nonsense!
The complex numbers for a complete metric space.  I'm not going to go into details about what this means, but it implies that if $\lim_{z\to a} f(z)$ exists, then it must be a complex number[2].  Therefore, per the definition written above, it is not possible for $\lim_{z\to a} f(z) = L$ to exist, but for $L$ to not be a complex number.  As such, the notation $\lim_{z\to a} f(z) \not\in \mathbb{C}$ is confusing and ambiguous.  On the one hand, it asserts that the limit exists.  On the other hand, it asserts that the limit is not a complex number.  These two statements contradict each other, which is confusing.  Thus it is best to avoid this notation.

[1] ...and then we learn more mathematics, learn about the extended real numbers, the Riemann sphere, the Alexandrov compactification, and other topological ideas which cure this overloading, but that is neither here nor there.
[2] As I have defined the limit, if $|f(z)|$ grows without bound as $z \to a$, then the limit does not exist.  In this case, we might write $\lim_{z\to a} f(z) = \infty$ and say that the limit is infinite.  However, per the definition written above, the limit does not exist.
A: It's possible to extend complex numbers into more than one complex dimension (so you could have $a+bi+cj+\ldots$) using quaternions, the set of which is written with $\mathbb{H}$. I'm not sure if this is exactly what your teacher is referring to, but it is true that your notation falls apart if the function is defined in terms of quaternion variables, as it is possible to have a non-complex quaternion number.
While it's unusual, I don't see anything wrong with your notation in casual usage (in 12th grade calculus I doubt you'll have to deal with quaternions), but if you want to be mathematically rigorous, there is at least one more number set which extends complex numbers that I'm aware of, so your notation is not technically correct.
