Why is continuity characterized with open sets? Why is the topological definition of continuous in terms of open sets?
I think my main complaint might be that the notion of open set seems too flexible/general and considers too many things that don't seem the right notion of "closeness". Conceptually, people explain "continuous" as:

Nearby points map to nearby points.

But we can easily construct sets for which *all their points are not “nearby” but they are still open. A simple example in metric spaces: the union of two open balls. The sets are still open but the points in one ball vs the other are not nearby. However the topological definition is in terms of open sets so it would consider maps balls like this from $X$ to $Y$ while that doesn't seem right to me. Is there something that I am missing? 
I guess I find it better to have a notion that captures the idea of “balls of radius epsilon in Y” to “balls of radius delta in X” a better notion of continuous. 
Another issue I find with this is that I find this in conflict even with the traditional epsilon-delta definition. The way I see it is that the topological definition should be more general (and abstract) and should encompass the metric space definition as a special case. Which to me it’s not clear it does because there is this union of disjoint open sets issue, that seem get included in the topological definition but for me they shouldn’t. This point seems important. Why were open sets chosen as the correct notion? A better definition for me would be (instead of open sets) to be in terms of “balls of radius epsilon in Y” to “balls of radius delta in X” in some topological way to define this. 
I have of course read the descriptions of open sets in wikiepdia but that doesn't seem to really clarify things. I know that open sets are the set of points under some topology that are "close". i.e. we only need sets to classify what points are considered "close". Which seems to me the main motivation why open sets were chosen, but the fact that disjoint open balls pass the test and are considered "close by" particularly disturbs me for some reason. Why is this specific complaint OK to ignore? What justifies not being worried about it?
Another reason I find it weird to use open sets is because for me open sets (since I am most familiar with the definition of open sets in metric spaces), are a type of set where everything is an interior point. It's a type of set that:

for all points we can always find a perturbation such that the point remains in the set (thus there is a neighbourhood that contains it in E). 

I find this problematic since it doesn't seem the right notion of "nearby" (at least to me); the reasons I prefer the definition to be restricted to only single open balls or sets that have no weird gaps (continuous sets? for some definition of that). This interior point issue doesn't seem to be what continuity (or limits actually) encompass conceptually. Continuity/limits seem to be a property about getting closer and closer (at least conceptually) or approaching. Therefore, for me it would be better to define it in terms of sets that reflected this idea of closeness. Something like neighbourhoods or (open) balls like in the traditional way of defining balls $B_{\delta}(p) = { x \in X | d(x,p) < \delta}$. Since this seems to be a clear notion of "nearby". Why are these ideas not preferred? What is wrong with it?
 A: I'm not sure why open balls are any better than open sets at capturing the notion of "closeness". If a ball has radius 1000000, the points in it won't be close in any sense. What you seem to be thinking of is not a particular open ball, but rather the set of all open balls around a point $p$. This more accurately captures the $\epsilon-\delta$ definition, as in that definition you are allowed to make $\delta$ as small as you like. Thus you may choose the open ball so that the points in it are arbitrarily near to $p$. Now, you can also do this for open sets, since every open ball is an open set.
This is formalized by the idea of a neighborhood basis, defined as any collection $N_p$ of open neighborhoods of $p$ which are ``arbitrarily small'' in the sense that if $U$ is any open set containing $p$, then there is a set in $N_p$ which is a subset of $U$. If the topological space is a metric space, we can take $N_p$ to be the set of all balls centered at $p$.
By making the neighborhoods become smaller and smaller, we can recover the idea of closeness encapsulated by the $\epsilon-\delta$ definition. To be precise, suppose $X$ and $Y$ are metric spaces, and let $p$ be a point of $X$. Choose an arbitrarily small element of the neighborhood basis of $f(p)$, that is, a ball $B_\epsilon(f(p))$ for arbitrarily small $\epsilon$. If $f$ is continuous in the topological sense, then $f^{-1}(B_\epsilon(f(p))$ is open, and it contains $p$, so it must contain some open ball $B_\delta(p)$. To say that $f(B_\delta(p))\subseteq B_\epsilon(f(p))$ means that if $|p_1-p|<\delta$, then $|f(p_1)-f(p)|<\epsilon$, which is the $\epsilon-\delta$ definition of continuity.
This shows that we can recover the $\epsilon-\delta$ definition by only applying the topological definition to those open subsets of $Y$ which happen to be open balls. Conversely, suppose we know that $f^{-1}(B_\epsilon(q))$ is open for all balls $B_\epsilon(q)$ in $Y$. Then it must follow that $f^{-1}(V)$ is open for every open set $V$. To see this, write $V=\bigcup_\alpha B_\alpha$ as a union of balls $B_\alpha$ (which we can do since $V$ is open). Then $f^{-1}(V)=\bigcup_\alpha f^{-1}(B_\alpha)$. By hypothesis, each $f^{-1}(B_\alpha)$ is open, therefore $f^{-1}(V)$ is also open.
Thus we could give an alternative characterization of continuity by simply requiring that $f^{-1}(V)$ be open when $V$ is an element of the neighborhood basis. However, if for $q\in Y$ we take $N_q$ to be the set of all open sets containing $q$, this is also a neighborhood basis, and so we see that this isn't so different from the usual definition.
I will remark that for a metric space, there is also a countable neighborhood basis of any point $p$, obtained by considering only those balls $B_\delta(p)$ where $\delta\in\mathbb Q^+$. This is essentially what allows us to express continuity in terms of limits, as in stressed out's answer.
Finally, let me address the issue of disjoint open sets. Suppose $V$ is an open neighborhood of $f(p)$ in $Y$ such that $f^{-1}(V)$ is a union of two disjoint open sets $U_1$ and $U_2$ in $X$, where, say, $p\in U_1$. If we consider $U_1$ to be the "nearby" points, all this says is that there are also some "non-nearby" points (in $U_2$) which map into $V$, which is not a problem.
A: You seem to have a misconception about what the notion of openness is supposed to represent. You write:

I know that open sets are the set of points under some topology that are close. i.e. we only need sets to classify what points are considered close. Which seems to me the main motivation why open sets were chosen, but the fact that disjoint open ball pass the test and are considered close by particularly disturbs me for some reason. Why is this specific complaint ok to ignore? What justifies not being worried about it?

It's not true that an open set is a set of points that are "close together". Open sets do not, and are not supposed to, be sets of points that are "all close to each other". That's why it's fine a union of disjoint open balls is considered an open set: contrary to what you write here, disjoint open balls are not "considered close by".
It looks like this is at the core of your confusion. Maybe things will make more sense now?
A: I think what User Randall wrote in a comment is the main point: Only half of the emphasis in the definition of continuity as

The inverse images of all open sets are open

should lie on

The inverse images of all open sets are open

but at least half of it on

The inverse images of all open sets are open.

The intuition is that a set is open if around each point inside, there is still some wiggle room a.k.a. neighburhood around it. Granted that some open sets in a metric space also contain some points "far away", as in your example with the disjoint union of two balls -- but now that is where the all in the definition kicks in: To check continuity, you will also have to consider single balls. Really, really small single balls. All of them.
And in a metric space, it is clear that checking it for "very small" balls, small as in "your favourite $\epsilon$", suffices to prove it for all. Without a metric, it's harder to tell which open sets are small, so, well, let's just make the definition robust and demand it for all of them. (Actually, sometimes it suffices to check on various kinds of "basic" open sets.)
So the "all" is a placeholder for arbitrarily small, which technically does not make sense in a general topological space. As soon as it does -- in a metric space --, you can replace "all" by "arbitrarily small", and making that rigorous will give the usual $\epsilon-\delta$-definition back.
A: Not every topological space is metric so the notion of distance is out of question for general topological spaces thus we can not measure if two points are close to each other or not.
Open sets are there to generalize the concepts such as continuity to all topological spaces without  applying the distance and $\epsilon-\delta$ terminology. 
If the inverse image of open sets are open then the function is continuous, is much easier than " For every epsilon there exists a delta....."    
A: In analogy to metric spaces, let's use the (not-formally defined) term "ball around $x$" to mean the open set of points which are close to $x$ by some reasonable notion of closeness. The stricter your notion of closeness, the smaller the ball. If you want something more formal, you can take "balls" to be elements of a basis for the topology, if you've heard of that. The right intuitive notion of "$U$ is an open set" is not "$U$ is a ball" but rather "$U$ is a union of balls."
Now let's look at the definition of continuity. "$f$ is continuous at $x$ if for all open $U \ni f(x)$, we have $f^{-1}(U)$ open." The first thing to note is that if $f^{-1}(U)$ and $f^{-1}(V)$ are both open, then so is $f^{-1}(U \cup V) = f^{-1}(U) \cup f^{-1}(V)$. So the definition holds if it works for all balls, since an arbitrary open set is a union of balls. When you restrict your attention to balls, this is exactly the topological analogue of the epsilon-delta definition in metric spaces. 
A: The most intuitive definition of continuity is indeed

nearby points map to nearby points

On the other side open subsets can be crazily complicated, and as already noted in other responses, they may contain points arbitrarily "far". In fact any two different points in a Hausdorff topology are "far" in the sense that they have disjoint neighborhoods. Unless you have a metric, all points are equally "far" one another.
The key observation is that the most direct formalization of the above definition is using neighborhoods, so

The inverse image of any neighborhood of $f(x)$ includes a neighborhood of $x$

From neighborhoods to topology
Given the full map $\mathfrak{N}: S \to \wp(\wp(S))$ of neighborhoods of points of a set $S$, such that

  
*
  
*$\forall x \in S, \forall N\in \mathfrak{N}[x] \quad x\in N$
  
*$\forall x \in S, \forall N\in \mathfrak{N}[x], \forall B\in\wp(S) \quad N \subseteq B \implies B\in \mathfrak{N}[x]$
  
*$\forall x \in S, \forall M\in \mathfrak{N}[x], \forall N\in \mathfrak{N}[x] \quad M\cap N \in \mathfrak{N}[x]$
  
*$\forall x \in S, \forall N\in \mathfrak{N}[x] \quad \exists M \in\mathfrak{N}[x] : M \subseteq N \land \forall y \in M: N \in \mathfrak{N}[y]$
  

you can very well define a topology on a space $S$, by defining open sets as those that contain neghborhoods of all their points:

$T = \{A \in \wp(S) : \forall x \exists N (N \in \mathfrak{N}[x] \land (x\in A \implies x\in N \subseteq A)) \}$

With this definition, it turns out that inverse images of open sets by a continuous function are open.
From topology to neighborhoods
As one can see, a "neighborhoodogy" is quite a complicated beast, a function that maps from points to "parts of parts of the space", subject to quite complex rules. On the other hand, a topology is much simpler, it's just a single subset of "parts of the space" $T \subseteq \wp(S)$, again with some rules, but less so.

  
*
  
*$\emptyset \in T$
  
*$S \in T$
  
*$\forall \mathcal{F}\subseteq T \quad \bigcup\mathcal{F}\in T$
  
*$\forall P\in T, \forall Q\in T \quad P\cap Q\in T$
  

We can then define neighborhoods of a point with

$\mathfrak{N}[x] = \{N \in \wp(S) : \exists O(O\in T \land x\in O \subseteq N)\}$

and continuous functions as those for which inverse images of open sets are open.
It turns out that the same property for neighborhoods are preserved, so the two definitions are completely equivalent. And the one based on topology is much simpler, requiring less concepts, less rules and less steps.
This is why it's commonly used as the preferred definition.
A: I personally have always had problem with the definition of continuity with respect to open sets. And my issue is not that open sets are too general, but my issue is that the definition looks sort of twisted and counter-intuitive to what one would expect it to be. 
If $f: X\to Y$ is a mapping between two topological spaces $X$ and $Y$, then $f$ is continuous if and only if $f^{-1}(U)$ is an open subset of $X$ for any $U$ that is open is $Y$. In metric spaces, we know that a set is open if we can find a ball of small enough radius centered around each of its points. So, one can easily see that this definition does imply the good old $\epsilon-\delta$ definition of continuity that we were taught in calculus.
But still, the idea of working with inverse images looks mysterious to me. A more natural looking definition, which is equivalent, is the definition that uses the idea of closures in topology. $f$ is continuous if and only if 
$$f(\mathrm{cl}(S)) \subseteq\mathrm{cl}(f(S))$$
is true for any set $S \subseteq X$. First of all, it looks more natural because we're working with $f$ and not $f^{-1}$. Secondly, a closure of a set $S$ is the set of all points that are arbitrarily "close" to it and therefore, this statement says that $f$ is continuous if and only if it sends close points to an arbitrary set $S$, to points that are close to its image. 
If you are in a good topological space like a metric space, you can define the closure of a set using the idea of the limit of a sequence. A point is in the closure if there exists a sequence that approaches toward it. Then $f$ can be equivalently defined as being continuous if and only if 
$$\lim_{n\to\infty}f(a_n)=f(\lim_{n\to\infty}a_n)$$
This is also more familiar and in essence is very similar to saying that $f$ maps nearby points to nearby points. 
Now, there are at least four axiomatic ways to define abstract topology: using the properties of open sets, using the properties of closed sets, using the properties of neighborhoods and using the properties of the closure operator which is due to Kuratowski if I'm not mistaken. It can be very instructive to check that all of these axiomatic systems turn out to be equivalent and see how continuity is defined in each of them.
Metric spaces in analysis are a special case of the first system of axioms for topology (open sets) where our space has a basis of open balls, because the definition of an open ball is very straightforward when we have a metric on $X$. 
A: It appears that most of your confusion comes from your notion of continuity being tightly coupled to the notion of a metric space. But in fact the notion of continuity for metric spaces is tremendously natural, and it is not necessary to define concepts like open balls or open sets to get at continuous functions over metric spaces.
However for functions between non-metric spaces, one cannot simply decide about what being "close" or "near" means, or even define what a "ball" is. It is in these cases that topologies and open sets shine.
One should note that there isn't anything special about open sets vs closed sets. It is perfectly reasonable to define topologies and continuity in terms of closed sets. (And in fact it is sometimes easier, such as in algebraic geometry).
Relatedly, you also seem to conflate the idea of two points being close and two points being contained in a single open set (in particular, you seem to dislike it when this open set is formed as a union of disjoint open balls in a metric space). But in fact any two points in any topological space are contained in an open set: namely the whole space (which is open). The intuition that being in an open set somehow means that two elements are close is a very dull intuition, and should be sharpened.
It is somehow better to think of two points as being close if they are contained in "lots" of open sets. And still this intuition leaves room for more careful examination.
A: 
I guess I find it better to have a notion that captures the idea of “balls of radius epsilon in Y” to “balls of radius delta in X” a better notion of continuous.

Your own Wikipedia link provides part of the answer:

In practice, however, open sets are usually chosen to provide a notion of nearness that is similar to that of metric spaces, without having a notion of distance defined. In particular, a topology allows defining properties such as continuity, connectedness, and compactness, which were originally defined by means of a distance.

