Is there an area of study regarding why certain mathematical definitions are useful? Often in my studies I'll come across an definition, which I understand, and  but don't necessarily see why the particular definition was chosen to be studied.  For example, the topological axioms (open sets, etc) have historically proven to be useful, and I can see them as useful in my studies, but what about these axioms made them so useful?  Or why, when a structure is defined, is it always the next step to study the maps which preserve that structure (vector spaces - linear functions, topological spaces - homeomorphisms, etc)? Why would a definition as clunky as Lp yield such elegant results? It's proved to be very difficult for me to find an answer to these types of questions.  Is there an area of study or a resource that would give me some answers to questions of this nature? 
 A: This is a very good question. The short answer is no, there is not a particular area of metamathematics that studies why certain mathematical definitions are useful. Indeed, the reasons why definitions of continuity and open sets and other definitions within topology are useful is exactly what the field of topology is. 
If you near or around mathematical historians, you might ask them. Understanding why certain definitions prevailed is deeply intertwined with the history and directions of mathematics itself.
We should also note that mathematics does not typically go in the order that you suggest. That is, one does not typically first define an open set and a continuous map and then see what happens. This is how textbooks and courses are organized, because it is a reasonable pedagogical order that mimics how we should think when we encounter new ideas. How does it fit with what we know? and Is this the same as something else I know? are fundamental questions that we ask and try to answer constantly.
Instead, consideration of things like the open set and continuous maps developed around the same time that things like calculus were being formalized and properly understood. Their current presentations have been distilled over decades of use, and there has been much effort in parallel presentation and comparison. But at the time of their first development, they would have looked very different. What we now think of as the axioms of topology are not yet 100 years old (or, perhaps appeared in an early form in 1914 by Hausdorff). Much of what we now think of as very abstractly topological was done in the context of maps between metric spaces (where the idea of continuity is much more clear and obvious).
Understanding how mathematics went from there to here might be enlightening, but it would also probably be convoluted. You might check out this paper on the history of topology, for instance.
A: Not really. You can try to get some answers here or on Quora but it's an uphill battle. 
I think the way most people deal with this sort of thing is to just keep using the definitions until they become second nature, in line with von Neumann's famous "in mathematics you don't understand things. You just get used to them." But I think this is pretty unsatisfying, and in particular I think it's possible to actually understand things and not just get used to them. 
I think this is particularly unsatisfying in the case of topological spaces. Among other things, one might have guessed that the correct notion of structure-preserving map between topological spaces is an open map, which it's not, and there's something to be explained about why the actual definition of continuous is correct beyond the fact that it generalizes continuity for functions between metric spaces. The short answer is that topological spaces are absurdly general; so general that they don't axiomatize topology in the intuitive sense but a kind of logic. The long answer is this book. 
Another thing you can do is to look up the history - find out who wrote down the definition and why they did it. 
A: I'd like to give somewhat of an example as to the way of most naturally developing, for instance, the axioms for a topological space. Admittedly this is most likely going to be an idealized path historically speaking (the insights are only clear after you've had them) but it gives an idea of exactly what topology generalizes. This should be able to then apply to groups, etc.
So in the beginning, people studied $\mathbb R^2$ and $\mathbb R^3$. There were various definitions and theorems that went along with the geometry and algebra of these spaces, and some mathematician at some point realized that the notions we discussed could typically be generalized to higher, less intuitive dimensions. Thus $\mathbb R^n$ is born.
With the offshoot of analysis, people begin to see that one of the most important definitions pertaining to this now-abstract space is the distance structure it holds, namely the function $D:\mathbb R^n\to\mathbb R$ defined by $$D(\vec x,\vec y)=\sum_i(x_i-y_i)^2.$$ This distance structure essentially gives rise to much of what analysis is (of course, field structures are also important, but that's an algebraic problem). And so people generalized this even more to define metric spaces, a set with a distance function defined only by three simple properties of $D$ (symmetry, positivity, and triangle inequality) that made precise the notion of generalized distance.
In the study of metric spaces, people begin to pick out a fundamental notion in this more general study as well - the notion of continuity. They realized through a trivial theorem that continuity can be defined purely in terms of open sets, and thus they went to generalize further to the theory of open sets, once again picking up three fundamental axioms governing their structure. And lo and behold, there is the theory of topological spaces.
