What's the advantage of defining topologies on open sets rather than closed sets? Munkres say in his book that in topology we could have defined topologies on closed sets and introduce that first before open sets. 
What's the advantage of defining topologies on closed sets rather than open ones?
Also does this mean limits can be defined as $$\forall \epsilon > 0, \exists \delta > 0$$ such that $$|x - a| \leq \delta \implies |f(x) - f(a)| \leq \epsilon$$
What happens if I change it to 
$$|x - a| < \delta \implies |f(x) - f(a)| \leq \epsilon$$
 A: To your first question:

What's the advantage of defining topologies on closed sets rather than open ones?

None, really; the two approaches are precisely equivalent.  It's a bit analogous to saying that you can define "even numbers" as integers of the form $2k$ for $k \in \mathbb{Z}$, and then define "odd numbers" as integers that are not even;  or you can define "odd numbers" as integers of the form $2k+1$ for $k \in \mathbb{Z}$, and then define "even numbers" as integers that are not odd.  It really doesn't matter, neither approach is "right", and it's ultimately a matter of taste and preference.  (This analogy is a very crude one, because in topology there are sets that are neither open nor closed, and sets like that don't correspond to anything in my even/odd analogy.)
To the second question:

Also does this mean limits can be defined as $$\forall \epsilon > 0, \exists \delta > 0$$ such that $$|x - a| \leq \delta \implies |f(x) - f(a)| \leq \epsilon$$

No, it doesn't mean that at all.  Munkres isn't saying that you can just replace open intervals with closed intervals throughout the entire theory and leave everything else unchanged and it will all work out the same. What he is saying is that you can take closed sets as the fundamental building blocks of the theory, but if you do that, you may have to make other changes too.  For example, in a theory based on open sets, we have the property that an arbitrary union of open sets is open; in a theory based on closed sets, we have the property that an arbitrary intersection of closed sets is closed.
A: Yes, we could do that. If you know all closed subsets of a topological space $X$, you can define that topology by just saying "Let the set of closed subsets of $X$ be ..." 
For example one can say: The usual topology on $\mathbb{R}$ contains the sets whose all points are interior as opens. Or equivalently: the usual topology on $\mathbb{R}$ contains the sets whose all points are limit points as closeds.
Then why do we choose the first one? Because most of the time it is hard to give all open or closed sets on a topology. Instead, we know a set called a basis, whose elements' arbitrary unions generate the open sets. Actually, we could define a set -let's call it cbasis- whose elements' arbitrary intersections generate the closed sets. But that cbasis would usually seem ugly (in my opinion).
Well, then Munkres is right. We can define a topology by giving the set of closed sets or by giving a cbasis.
A: I decided to expand on my comment...
The general definition of a topology $\tau$ on some space $X$ is by specifying a family of subsets of $X$ with the following axioms:


*

*$\emptyset \in \tau$ and $X \in \tau$

*For any sub-family of sets $\psi \subset \tau$ we have $\bigcup_{S \in \psi} S\in \tau$

*For any finite sub-family of sets $\phi \subset \tau$ we have $\bigcap_{S \in \phi} S \in \tau$


Now we call these sets open sets, and define the closed sets to be the compliment of any open set. Equivalently we could define $\tau$ to be some family of subsets of $X$ with the following axioms:


*

*$\emptyset \in \tau$ and $X \in \tau$

*For any sub-family of sets $\psi \subset \tau$ we have $\bigcap_{S \in \psi} S\in \tau$

*For any finite sub-family of sets $\phi \subset \tau$ we have $\bigcup_{S \in \phi} S \in \tau$


and then call these the closed sets and an open set to be a compliment of any closed set.
Now why would we consider defining the topology the first way as opposed to the second way. Well the definition of a neighborhood of some point $x \in X$ is some subset $N$ of $X$ so that $\exists B$ an open set where $B \subset N$ and $x \in B$. So in defining the topology in the first sense we can simply say that $N$ is a neighborhood of $x$ if $\exists B \in \tau$ where $B \subset N, x \in B$.
Of course one could ask well why is the neighborhood defined this way, and I personally cannot answer that question. From my perspective it's one of those things that is defined in a way so that it's useful and easy to work with. It may in fact just be a historical reason as to why everything is oriented around open sets as opposed to closed sets (since anything defined in terms of open sets can equivalently be defined in terms of closed sets).
Now for an example of a topology specified, in some sense, in terms of the closed sets let's look to the cofinite topology of some space. Give me some infinite space $X$, let's define the closed sets to be $X$ or any set with a finite number of elements, i.e.
$$
\mathcal{C} = \{ X \} \cup \{ A \mid A \text{ is finite} \}
$$
Now from here we can define the topology in the usual sense as
$$
\tau = \{ A \mid A = \emptyset \text{ or } X \setminus A \text{ is finite} \}
$$
A: This is more an observation than anything else, but the Zariski topology on the set of prime ideals of a commutative ring $R$ is defined via closed sets. For any ideal $I$ of $R$ one defines the closed set $V(I)$ of prime ideals that contain $I$ and go from there. Most any book on commutative algebra, and some on abstract algebra, will have more details.
