There is a reason set theory does not use the symbol $\infty$ - because it would be very easily confused by the way it has been used in limits and calculus in general for a long time. In particular, limits are talking about something that can best be called "linear infinity" or "topological infinity."
Essentially, we can add "values" $-\infty$ and $+\infty$, to the real line, and give them meaning in the sense of convergence, without making them numbers. There is a deep sense in which convergence to these new values is "the same as" convergence to other values on the real line. But the real key is that these two values are not numbers.
The sequences $\{\frac{1}{n}\}$ and $\{e^{-n}\}$ both converge to $0$ at very different rates, but we don't call those zeros different. How fast something converges can be wildly different, but that doesn't mean that the value to which they converge is different.
There is a sense in which convergence can be faster and slower, but that has nothing to do with infinity. You can do the same example above with:
$$\lim_{x\to 0} x = 0 =\lim_{x\to 0} e^{-1/x^2}$$
More generally, $f(x)\to+\infty$ is exactly the same as $\arctan f(x)\to\frac{\pi}{2}$, and similarly, $f(x)\to-\infty$ is exactly the same as $\arctan f(x)\to-\frac{\pi}2$. So the idea that there are different "values" of infinity in calculus is a bit of confusion. In this sense, convergence to "infinity" is the same as convergence to $1$ of a function with range the interval $(-1,1)$.
One typical way to define the real numbers is to define the notion of a Dedekind cut on the rationals, $\mathbb Q$. A Dedekind cut is a subset $C$ of $\mathbb Q$ with the following properties:
- $\forall x\in C\,\forall y\in\mathbb Q(y<x\implies y\in C)$
- $\forall x\in C\,\exists y\in C(x<y)$
- $C\neq \emptyset$
- $C\neq \mathbb Q$.
The set of real numbers is defined as the set of all Dedekind cuts.
But if we defined an Extended Cut with just (1) and (2), then $\emptyset$ corresponds to $-\infty$ and $\mathbb Q$ corresponds to $+\infty$. So in this sense, having the extended real line is "simpler" than having just the real line.
The main reason we start with Dedekind cuts is our obsession with the algebraic properties of $\mathbb R$. We can't define addition on the extended real line in a useful way, for example, because of the problem of $+\infty+(-\infty)$. But if we were only interested in the topological properties of the real line (continuity, limits, etc) then the extended real line actually makes more sense, and it has the advantage that it is simpler.